2024-09-15T02:30:02.226 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T02:30:02.238 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:30:02.310 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905167 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: skanta@redhat.com first_in_suite: false job_id: '7905167' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANFhU/a/CUPXrEWTVV20x7hzW6BexpA+5ACSoNhbCX27dt7CCZt/wIdl4MsonDmmviovkkvzPzukmkYBubWunE= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T02:30:02.311 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T02:30:02.311 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T02:30:02.312 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T02:30:02.314 INFO:teuthology.task.internal:Checking packages... 2024-09-15T02:30:02.335 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T02:30:02.335 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T02:30:02.335 INFO:teuthology.packaging:ref: None 2024-09-15T02:30:02.335 INFO:teuthology.packaging:tag: None 2024-09-15T02:30:02.335 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:30:02.335 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:30:02.335 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:30:02.487 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T02:30:02.489 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T02:30:02.495 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T02:30:02.495 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T02:30:02.552 INFO:teuthology.task.internal:Saving configuration 2024-09-15T02:30:02.561 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T02:30:02.628 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T02:30:02.657 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:25:14.201782', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANFhU/a/CUPXrEWTVV20x7hzW6BexpA+5ACSoNhbCX27dt7CCZt/wIdl4MsonDmmviovkkvzPzukmkYBubWunE='} 2024-09-15T02:30:02.657 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T02:30:02.736 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-15T02:30:02.736 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T02:30:02.812 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8a62983ac0>, signals=[15]) 2024-09-15T02:30:02.813 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T02:30:02.861 INFO:teuthology.task.internal:Opening connections... 2024-09-15T02:30:02.861 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-09-15T02:30:02.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:30:02.939 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T02:30:02.944 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-09-15T02:30:03.019 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-09-15T02:30:03.019 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-09-15T02:30:03.077 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-09-15T02:30:03.077 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-09-15T02:30:03.077 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-09-15T02:30:03.077 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-09-15T02:30:03.077 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-09-15T02:30:03.077 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-09-15T02:30:03.077 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T02:30:03.078 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-09-15T02:30:03.078 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-09-15T02:30:03.078 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T02:30:03.078 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-09-15T02:30:03.078 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T02:30:03.078 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T02:30:03.078 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T02:30:03.079 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-09-15T02:30:03.100 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T02:30:03.108 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T02:30:03.153 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T02:30:03.153 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:30:03.170 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T02:30:03.253 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T02:30:03.253 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:30:03.341 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T02:30:03.372 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T02:30:03.385 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T02:30:03.385 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T02:30:03.386 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:30:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:30:03.403 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-15T02:30:03.461 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-503.el9.x86_64 2024-09-15T02:30:03.462 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-503.el9.x86_64 2024-09-15T02:30:03.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-09-15T02:30:05.775 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-15T02:30:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-15T02:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-15T02:30:13.869 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-15T02:30:14.847 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-15T02:30:20.345 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-15T02:30:22.359 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T02:30:22.360 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T02:30:22.446 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:30:22.452 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:30:22.452 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-15T02:30:22.452 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:30:22.452 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:30:22.452 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-09-15T02:30:22.453 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:30:22.454 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 79 M 2024-09-15T02:30:22.454 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 126 M 2024-09-15T02:30:22.454 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:30:22.563 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 258 kB/s | 23 kB 00:00 2024-09-15T02:30:22.938 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-15T02:30:23.338 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 40 MB/s | 35 MB 00:00 2024-09-15T02:30:24.063 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 19 MB/s | 29 MB 00:01 2024-09-15T02:30:24.064 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:30:24.064 INFO:teuthology.orchestra.run.smithi137.stdout:Total 49 MB/s | 79 MB 00:01 2024-09-15T02:30:24.751 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:30:25.100 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:30:25.100 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:30:26.071 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:30:26.071 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:30:28.515 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:30:29.438 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T02:30:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:30:32.319 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:30:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:30:37.270 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:30:37.308 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:30:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:31:20.436 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:31:20.477 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:31:20.810 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:31:20.810 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T02:31:20.810 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T02:31:20.810 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T02:31:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T02:31:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:31:21.663 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:31:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T02:31:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T02:31:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T02:31:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T02:31:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:31:21.664 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:31:22.024 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T02:31:22.112 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T02:31:22.112 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T02:31:22.112 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T02:31:22.113 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:31:22.113 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:31:22.113 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T02:31:22.113 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:31:22.113 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T02:31:22.172 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T02:31:22.172 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T02:31:22.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-09-15T02:31:22.213 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-09-15T02:31:22.213 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-09-15T02:31:22.213 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 82.el9 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8274670 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-09-15T02:31:22.214 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:31:22.215 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T02:31:22.216 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T02:31:22.216 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-505.el9.x86_64 2024-09-15T02:31:22.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T02:31:22.869 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-09-15T02:31:24.186 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T02:31:24.216 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-09-15T02:31:24.218 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-15T02:31:24.234 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.Mf0Ojgo3mz 2024-09-15T02:31:24.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Mf0Ojgo3mz 2024-09-15T02:31:24.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.Mf0Ojgo3mz 2024-09-15T02:31:24.450 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.Mf0Ojgo3mz is 6KB 2024-09-15T02:31:24.471 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.Mf0Ojgo3mz 2024-09-15T02:31:24.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T02:31:24.549 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T02:31:24.549 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T02:31:24.549 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T02:31:24.549 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T02:31:24.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T02:31:24.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-09-15T02:31:54.743 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T02:31:54.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-15T02:31:54.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:32:54.805 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T02:33:03.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-15T02:33:03.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:33:04.244 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-15T02:33:04.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-15T02:33:04.606 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T02:33:04.607 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-15T02:33:04.621 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-505.el9.x86_64 2024-09-15T02:33:04.622 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T02:33:04.622 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:33:04.622 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T02:33:05.622 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T02:33:05.623 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:33:05.701 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-09-15T02:33:05.746 DEBUG:teuthology.parallel:result is None 2024-09-15T02:33:05.746 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T02:33:05.754 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T02:33:05.754 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:33:05.770 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T02:33:05.891 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T02:33:06.024 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T02:33:06.024 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:33:06.043 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T02:33:06.133 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T02:33:06.133 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:33:06.161 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:33:06.171 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:33:06.173 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T02:33:06.191 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T02:33:06.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:33:06.241 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T02:33:06.250 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T02:33:06.250 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:33:06.300 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:33:06.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:33:06.444 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:33:06.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:33:06.572 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-15T02:33:06.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:33:06.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-09-15T02:33:06.746 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T02:33:07.155 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T02:33:07.163 INFO:teuthology.task.internal:Starting timer... 2024-09-15T02:33:07.163 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T02:33:07.220 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T02:33:07.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-09-15T02:33:07.377 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-09-15T02:33:07.379 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T02:33:07.379 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-09-15T02:33:07.411 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-09-15T02:33:07.411 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-09-15T02:33:07.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T02:33:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:33:07.480 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T02:33:07.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-09-15T02:33:07.575 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T02:33:07.584 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T02:33:07.591 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T02:33:07.720 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T02:33:07.731 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T02:33:07.732 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T02:36:49.545 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-09-15T02:36:49.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-15T02:36:49.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:36:49.626 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-15T02:36:49.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-15T02:36:49.712 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T02:36:49.728 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T02:36:49.728 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:36:49.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:36:49.782 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T02:36:49.796 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T02:36:49.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-09-15T02:36:49.832 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-09-15T02:36:49.846 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T02:36:49.860 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T02:36:49.917 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:36:49.921 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:36:49.921 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-09-15T02:36:49.921 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:36:49.921 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:36:49.921 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:36:49.921 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:36:49.922 INFO:teuthology.run_tasks:Running task install... 2024-09-15T02:36:49.930 DEBUG:teuthology.task.install:project ceph 2024-09-15T02:36:49.930 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T02:36:49.930 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T02:36:49.930 INFO:teuthology.task.install:Using flavor: default 2024-09-15T02:36:49.939 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T02:36:49.939 INFO:teuthology.task.install:extra packages: [] 2024-09-15T02:36:49.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T02:36:50.518 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:44 ago on Sun 15 Sep 2024 02:36:06 AM UTC. 2024-09-15T02:36:50.605 INFO:teuthology.orchestra.run.smithi137.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T02:36:50.647 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:36:50.647 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:36:50.648 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:36:50.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo dnf -y copr enable ceph/el9 2024-09-15T02:36:50.892 INFO:teuthology.orchestra.run.smithi137.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T02:36:50.892 INFO:teuthology.orchestra.run.smithi137.stderr:of the main distribution, and quality may vary. 2024-09-15T02:36:50.892 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-15T02:36:50.892 INFO:teuthology.orchestra.run.smithi137.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T02:36:50.893 INFO:teuthology.orchestra.run.smithi137.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T02:36:50.893 INFO:teuthology.orchestra.run.smithi137.stderr:, 2024-09-15T02:36:50.893 INFO:teuthology.orchestra.run.smithi137.stderr:and packages are not held to any quality or security level. 2024-09-15T02:36:50.893 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-15T02:36:50.894 INFO:teuthology.orchestra.run.smithi137.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T02:36:50.894 INFO:teuthology.orchestra.run.smithi137.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T02:36:50.948 INFO:teuthology.orchestra.run.smithi137.stdout:Repository successfully enabled. 2024-09-15T02:36:50.982 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T02:36:50.982 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:36:51.128 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T02:36:51.128 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T02:36:51.239 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T02:36:51.239 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-15T02:36:51.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T02:36:51.265 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T02:36:51.265 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T02:36:51.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T02:36:51.407 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T02:36:51.472 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-09-15T02:36:51.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-09-15T02:36:51.700 INFO:teuthology.orchestra.run.smithi137.stdout:45 files removed 2024-09-15T02:36:51.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-09-15T02:36:52.269 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 300 kB/s | 77 kB 00:00 2024-09-15T02:36:52.552 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T02:36:52.828 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T02:36:53.004 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for el9 owned by ceph 54 kB/s | 7.6 kB 00:00 2024-09-15T02:36:55.745 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.2 MB 00:02 2024-09-15T02:37:16.161 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:19 2024-09-15T02:37:21.862 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-15T02:37:23.387 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-15T02:37:24.274 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-15T02:37:29.739 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-15T02:37:32.223 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:37:32.239 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:37:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:37:32.239 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:37:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T02:37:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T02:37:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T02:37:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T02:37:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T02:37:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T02:37:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T02:37:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T02:37:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:32.243 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:37:32.243 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:37:32.243 INFO:teuthology.orchestra.run.smithi137.stdout:Install 22 Packages 2024-09-15T02:37:32.243 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-09-15T02:37:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:32.244 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 58 M 2024-09-15T02:37:32.244 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:37:33.431 INFO:teuthology.orchestra.run.smithi137.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 6.9 MB/s | 5.0 MB 00:00 2024-09-15T02:37:33.473 INFO:teuthology.orchestra.run.smithi137.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 609 kB/s | 25 kB 00:00 2024-09-15T02:37:33.548 INFO:teuthology.orchestra.run.smithi137.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 8.3 MB/s | 637 kB 00:00 2024-09-15T02:37:33.632 INFO:teuthology.orchestra.run.smithi137.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 5.4 MB/s | 458 kB 00:00 2024-09-15T02:37:34.032 INFO:teuthology.orchestra.run.smithi137.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 11 MB/s | 14 MB 00:01 2024-09-15T02:37:34.116 INFO:teuthology.orchestra.run.smithi137.stdout:(6/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 10 MB/s | 4.9 MB 00:00 2024-09-15T02:37:34.132 INFO:teuthology.orchestra.run.smithi137.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 451 kB/s | 45 kB 00:00 2024-09-15T02:37:34.166 INFO:teuthology.orchestra.run.smithi137.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.3 MB/s | 120 kB 00:00 2024-09-15T02:37:34.192 INFO:teuthology.orchestra.run.smithi137.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.6 MB/s | 156 kB 00:00 2024-09-15T02:37:34.217 INFO:teuthology.orchestra.run.smithi137.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 6.2 MB/s | 320 kB 00:00 2024-09-15T02:37:34.242 INFO:teuthology.orchestra.run.smithi137.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 5.8 MB/s | 295 kB 00:00 2024-09-15T02:37:34.267 INFO:teuthology.orchestra.run.smithi137.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T02:37:34.501 INFO:teuthology.orchestra.run.smithi137.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 13 MB/s | 22 MB 00:01 2024-09-15T02:37:34.543 INFO:teuthology.orchestra.run.smithi137.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 533 kB/s | 160 kB 00:00 2024-09-15T02:37:34.601 INFO:teuthology.orchestra.run.smithi137.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 136 kB/s | 45 kB 00:00 2024-09-15T02:37:34.669 INFO:teuthology.orchestra.run.smithi137.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 633 kB/s | 42 kB 00:00 2024-09-15T02:37:34.694 INFO:teuthology.orchestra.run.smithi137.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-15T02:37:34.719 INFO:teuthology.orchestra.run.smithi137.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-15T02:37:34.769 INFO:teuthology.orchestra.run.smithi137.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-09-15T02:37:34.795 INFO:teuthology.orchestra.run.smithi137.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-15T02:37:34.819 INFO:teuthology.orchestra.run.smithi137.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 784 kB/s | 67 kB 00:00 2024-09-15T02:37:34.870 INFO:teuthology.orchestra.run.smithi137.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-15T02:37:35.004 INFO:teuthology.orchestra.run.smithi137.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 16 MB/s | 2.9 MB 00:00 2024-09-15T02:37:35.072 INFO:teuthology.orchestra.run.smithi137.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 11 MB/s | 3.1 MB 00:00 2024-09-15T02:37:35.077 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:37:35.077 INFO:teuthology.orchestra.run.smithi137.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-15T02:37:35.141 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:37:35.174 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:37:35.174 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:37:35.845 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:37:35.845 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:37:37.069 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:37:37.404 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T02:37:37.491 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:37:37.583 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:37:37.669 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:37:37.692 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:37:37.781 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:37:37.826 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T02:37:37.855 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T02:37:38.291 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T02:37:38.320 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:37:38.366 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:37:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T02:37:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T02:37:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:37:38.537 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:37:38.565 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:37:38.614 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:37:38.662 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T02:37:38.698 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T02:37:38.732 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T02:37:39.022 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T02:37:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:37:39.151 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:37:39.223 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T02:37:39.254 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T02:37:41.625 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:37:41.667 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:37:42.201 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:37:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:37:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:37:42.327 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T02:37:42.327 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T02:37:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:42.381 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:37:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-09-15T02:37:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:53.531 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:37:53.610 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:37:53.610 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:37:53.610 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T02:37:53.610 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:37:53.610 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:37:53.610 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:53.612 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:37:53.693 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:37:53.693 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T02:37:54.993 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T02:37:54.994 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T02:37:54.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T02:37:54.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:37:54.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T02:37:54.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T02:37:55.958 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:37:55.958 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:55.958 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T02:37:55.959 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.960 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:37:55.961 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T02:37:55.961 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:55.961 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:37:56.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-09-15T02:37:56.943 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:37:57.086 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:37:57.086 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:37:57.086 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:37:57.086 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-09-15T02:37:57.087 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:37:57.088 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 48 M 2024-09-15T02:37:57.088 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 193 M 2024-09-15T02:37:57.088 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:37:57.288 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-15T02:37:57.364 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 847 kB/s | 64 kB 00:00 2024-09-15T02:37:57.389 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 647 kB/s | 163 kB 00:00 2024-09-15T02:37:58.673 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-15T02:37:58.675 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:37:58.675 INFO:teuthology.orchestra.run.smithi137.stdout:Total 31 MB/s | 48 MB 00:01 2024-09-15T02:37:58.703 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:37:58.722 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:37:58.722 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:37:59.089 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:37:59.089 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:37:59.608 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:37:59.683 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T02:37:59.731 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T02:37:59.758 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:38:04.170 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:38:04.247 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:38:06.497 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:38:06.497 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T02:38:06.497 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T02:38:06.497 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:06.833 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:07.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-09-15T02:38:07.671 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:38 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T02:38:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T02:38:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:Install 34 Packages 2024-09-15T02:38:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 27 M 2024-09-15T02:38:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 96 M 2024-09-15T02:38:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:08.322 INFO:teuthology.orchestra.run.smithi137.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-15T02:38:08.506 INFO:teuthology.orchestra.run.smithi137.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.6 MB/s | 1.4 MB 00:00 2024-09-15T02:38:08.564 INFO:teuthology.orchestra.run.smithi137.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-15T02:38:08.665 INFO:teuthology.orchestra.run.smithi137.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.4 MB/s | 240 kB 00:00 2024-09-15T02:38:08.723 INFO:teuthology.orchestra.run.smithi137.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 9.8 MB/s | 3.9 MB 00:00 2024-09-15T02:38:09.015 INFO:teuthology.orchestra.run.smithi137.stdout:(6/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-15T02:38:09.040 INFO:teuthology.orchestra.run.smithi137.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 675 kB/s | 253 kB 00:00 2024-09-15T02:38:09.099 INFO:teuthology.orchestra.run.smithi137.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-15T02:38:09.149 INFO:teuthology.orchestra.run.smithi137.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-15T02:38:09.174 INFO:teuthology.orchestra.run.smithi137.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 671 kB/s | 106 kB 00:00 2024-09-15T02:38:09.208 INFO:teuthology.orchestra.run.smithi137.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 2.5 MB/s | 1.2 MB 00:00 2024-09-15T02:38:09.234 INFO:teuthology.orchestra.run.smithi137.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-15T02:38:09.509 INFO:teuthology.orchestra.run.smithi137.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 116 kB/s | 35 kB 00:00 2024-09-15T02:38:09.717 INFO:teuthology.orchestra.run.smithi137.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 245 kB/s | 51 kB 00:00 2024-09-15T02:38:09.742 INFO:teuthology.orchestra.run.smithi137.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 152 kB/s | 77 kB 00:00 2024-09-15T02:38:09.784 INFO:teuthology.orchestra.run.smithi137.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-15T02:38:09.809 INFO:teuthology.orchestra.run.smithi137.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.8 MB/s | 172 kB 00:00 2024-09-15T02:38:09.835 INFO:teuthology.orchestra.run.smithi137.stdout:(18/34): python3-mako-1.1.4-6.el9.noarch.rpm 261 kB/s | 172 kB 00:00 2024-09-15T02:38:09.852 INFO:teuthology.orchestra.run.smithi137.stdout:(19/34): python3-toml-0.10.2-6.el9.noarch.rpm 310 kB/s | 42 kB 00:00 2024-09-15T02:38:09.885 INFO:teuthology.orchestra.run.smithi137.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-09-15T02:38:09.910 INFO:teuthology.orchestra.run.smithi137.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 345 kB/s | 18 kB 00:00 2024-09-15T02:38:09.927 INFO:teuthology.orchestra.run.smithi137.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 578 kB/s | 23 kB 00:00 2024-09-15T02:38:09.944 INFO:teuthology.orchestra.run.smithi137.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 1.2 MB/s | 19 kB 00:00 2024-09-15T02:38:09.961 INFO:teuthology.orchestra.run.smithi137.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 592 kB/s | 20 kB 00:00 2024-09-15T02:38:09.986 INFO:teuthology.orchestra.run.smithi137.stdout:(25/34): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-09-15T02:38:10.003 INFO:teuthology.orchestra.run.smithi137.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-15T02:38:10.029 INFO:teuthology.orchestra.run.smithi137.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-15T02:38:10.054 INFO:teuthology.orchestra.run.smithi137.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-15T02:38:10.088 INFO:teuthology.orchestra.run.smithi137.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-09-15T02:38:10.113 INFO:teuthology.orchestra.run.smithi137.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-09-15T02:38:10.138 INFO:teuthology.orchestra.run.smithi137.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-15T02:38:10.163 INFO:teuthology.orchestra.run.smithi137.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-15T02:38:10.188 INFO:teuthology.orchestra.run.smithi137.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T02:38:10.214 INFO:teuthology.orchestra.run.smithi137.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-15T02:38:10.219 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:10.219 INFO:teuthology.orchestra.run.smithi137.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-15T02:38:10.354 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:10.390 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:10.391 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:10.763 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:10.763 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:12.022 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:12.092 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T02:38:12.124 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T02:38:12.165 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T02:38:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T02:38:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T02:38:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T02:38:12.411 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T02:38:12.441 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T02:38:12.475 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T02:38:12.509 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T02:38:12.549 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T02:38:12.588 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T02:38:12.622 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T02:38:12.670 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T02:38:12.702 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:38:12.776 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T02:38:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T02:38:12.994 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T02:38:13.055 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T02:38:13.094 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T02:38:13.142 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T02:38:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T02:38:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T02:38:13.431 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T02:38:13.510 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T02:38:13.615 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T02:38:13.655 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T02:38:13.770 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T02:38:13.927 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T02:38:13.957 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:38:13.983 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:38:13.983 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:38:13.983 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T02:38:13.983 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:38:13.983 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:38:13.983 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:15.359 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:38:15.393 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:38:15.393 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:38:15.393 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T02:38:15.393 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:38:15.393 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:38:15.393 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:15.769 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:38:15.792 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:38:15.793 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:38:15.793 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T02:38:15.793 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:38:15.793 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:38:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:16.003 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:38:16.026 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:38:16.027 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:38:16.027 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T02:38:16.027 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:38:16.027 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:38:16.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:16.092 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:38:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:38:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T02:38:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T02:38:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T02:38:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T02:38:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T02:38:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T02:38:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T02:38:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T02:38:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T02:38:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T02:38:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T02:38:18.820 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T02:38:18.820 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T02:38:18.821 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T02:38:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:18.823 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:19.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-09-15T02:38:19.789 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:50 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:19.879 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:38:19.921 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:19.922 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:38:19.922 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:19.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-09-15T02:38:20.563 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:20.694 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 83 k 2024-09-15T02:38:20.695 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 375 k 2024-09-15T02:38:20.696 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:20.885 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 436 kB/s | 83 kB 00:00 2024-09-15T02:38:20.886 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:20.886 INFO:teuthology.orchestra.run.smithi137.stdout:Total 433 kB/s | 83 kB 00:00 2024-09-15T02:38:20.886 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:20.891 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:20.891 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:20.893 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:20.893 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:20.987 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:21.434 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:38:21.530 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:38:21.872 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:38:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:38:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:22.107 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:22.108 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:22.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T02:38:22.896 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:23.031 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:23.032 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:23.032 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-15T02:38:23.032 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:23.032 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:23.032 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T02:38:23.032 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:23.032 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:23.033 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:23.033 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:38:23.033 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:23.033 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 137 k 2024-09-15T02:38:23.033 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 379 k 2024-09-15T02:38:23.033 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:23.249 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 637 kB/s | 137 kB 00:00 2024-09-15T02:38:23.249 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:23.250 INFO:teuthology.orchestra.run.smithi137.stdout:Total 632 kB/s | 137 kB 00:00 2024-09-15T02:38:23.250 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:23.257 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:23.257 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:23.313 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:23.314 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:38:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:38:23.567 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:38:23.567 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T02:38:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:24.471 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:38:24.471 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:24.471 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:24.471 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:24.471 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:24.471 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:24.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-09-15T02:38:25.270 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:25.360 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:38:25.402 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:25.403 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:38:25.403 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:25.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T02:38:26.047 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:26.182 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:26.182 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:26.183 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:26.184 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:26.184 INFO:teuthology.orchestra.run.smithi137.stdout:Install 7 Packages 2024-09-15T02:38:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:26.184 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.7 M 2024-09-15T02:38:26.184 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 80 M 2024-09-15T02:38:26.184 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:26.486 INFO:teuthology.orchestra.run.smithi137.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 163 kB/s | 24 kB 00:00 2024-09-15T02:38:26.511 INFO:teuthology.orchestra.run.smithi137.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 83 kB/s | 15 kB 00:00 2024-09-15T02:38:26.553 INFO:teuthology.orchestra.run.smithi137.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T02:38:26.578 INFO:teuthology.orchestra.run.smithi137.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T02:38:26.603 INFO:teuthology.orchestra.run.smithi137.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-15T02:38:26.629 INFO:teuthology.orchestra.run.smithi137.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T02:38:26.962 INFO:teuthology.orchestra.run.smithi137.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 5.4 MB/s | 3.4 MB 00:00 2024-09-15T02:38:26.966 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:26.966 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.7 MB/s | 3.7 MB 00:00 2024-09-15T02:38:26.966 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:26.974 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:26.974 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:27.068 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:27.068 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:27.374 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:27.438 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T02:38:27.464 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T02:38:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T02:38:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T02:38:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T02:38:28.564 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T02:38:28.617 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:38:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:38:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T02:38:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T02:38:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T02:38:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T02:38:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T02:38:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T02:38:29.898 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:29.899 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:30.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T02:38:30.689 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:01 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:30.823 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:30.824 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-09-15T02:38:30.824 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T02:38:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-09-15T02:38:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:30.827 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 38 M 2024-09-15T02:38:30.827 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 212 M 2024-09-15T02:38:30.827 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:31.170 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-15T02:38:31.229 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-15T02:38:31.262 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 355 kB/s | 33 kB 00:00 2024-09-15T02:38:31.304 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-15T02:38:31.365 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 634 kB/s | 38 kB 00:00 2024-09-15T02:38:31.515 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-15T02:38:31.574 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.7 MB/s | 3.0 MB 00:00 2024-09-15T02:38:31.715 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-09-15T02:38:31.766 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 977 kB/s | 245 kB 00:00 2024-09-15T02:38:31.791 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 3.3 MB/s | 250 kB 00:00 2024-09-15T02:38:32.150 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-09-15T02:38:32.233 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.7 MB/s | 4.9 MB 00:00 2024-09-15T02:38:32.234 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:32.234 INFO:teuthology.orchestra.run.smithi137.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-15T02:38:32.590 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:32.616 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:32.616 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:33.107 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:33.107 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:33.863 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:33.931 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T02:38:33.980 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:38:34.076 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:38:34.115 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T02:38:34.299 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T02:38:34.331 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T02:38:34.407 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T02:38:35.068 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T02:38:35.185 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T02:38:36.422 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:38:37.469 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T02:38:37.497 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:38:39.996 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:38:39.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T02:38:39.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:38:39.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:38:39.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T02:38:39.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T02:38:39.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T02:38:39.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T02:38:39.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T02:38:39.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T02:38:39.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:38:39.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T02:38:40.483 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T02:38:40.483 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:40.483 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:40.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:40.483 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:40.484 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:40.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-09-15T02:38:41.426 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:12 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:41.562 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:41.563 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:41.563 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:41.563 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:41.563 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:41.563 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T02:38:41.563 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:38:41.563 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:41.564 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:41.565 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:41.565 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-09-15T02:38:41.565 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:41.565 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.2 M 2024-09-15T02:38:41.565 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-09-15T02:38:41.566 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:42.380 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 287 kB/s | 47 kB 00:00 2024-09-15T02:38:42.738 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-15T02:38:42.772 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-15T02:38:43.413 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 215 kB/s | 222 kB 00:01 2024-09-15T02:38:43.562 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 193 kB/s | 159 kB 00:00 2024-09-15T02:38:43.607 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 334 kB/s | 279 kB 00:00 2024-09-15T02:38:43.640 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 417 kB/s | 32 kB 00:00 2024-09-15T02:38:43.673 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 218 kB/s | 14 kB 00:00 2024-09-15T02:38:43.698 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-15T02:38:43.724 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T02:38:43.749 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.6 MB/s | 90 kB 00:00 2024-09-15T02:38:43.782 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 9.4 MB/s | 1.0 MB 00:00 2024-09-15T02:38:44.033 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 87 kB/s | 54 kB 00:00 2024-09-15T02:38:44.037 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:44.037 INFO:teuthology.orchestra.run.smithi137.stdout:Total 916 kB/s | 2.2 MB 00:02 2024-09-15T02:38:44.108 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:44.124 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:44.124 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:44.211 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:44.213 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:44.722 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:44.905 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T02:38:44.959 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T02:38:45.022 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T02:38:45.059 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T02:38:45.100 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T02:38:45.203 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T02:38:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T02:38:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T02:38:46.060 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T02:38:46.108 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T02:38:46.149 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T02:38:46.195 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T02:38:46.219 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:38:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:38:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T02:38:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T02:38:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T02:38:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T02:38:47.307 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:47.308 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:47.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T02:38:48.138 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:19 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:48.270 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:48.271 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:48.271 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:48.272 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:48.273 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-09-15T02:38:48.273 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:48.273 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 7.0 M 2024-09-15T02:38:48.273 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-09-15T02:38:48.273 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:48.793 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-15T02:38:48.843 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T02:38:48.868 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 588 kB/s | 124 kB 00:00 2024-09-15T02:38:48.894 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T02:38:48.936 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-15T02:38:49.019 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-15T02:38:49.022 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:49.022 INFO:teuthology.orchestra.run.smithi137.stdout:Total 9.3 MB/s | 7.0 MB 00:00 2024-09-15T02:38:49.092 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:49.103 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:49.103 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:49.202 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:49.202 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:49.511 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:49.677 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T02:38:49.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T02:38:50.246 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T02:38:50.324 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T02:38:50.379 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T02:38:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:38:51.182 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:38:51.182 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T02:38:51.182 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T02:38:51.182 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T02:38:51.182 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T02:38:51.182 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T02:38:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:51.455 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:51.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-09-15T02:38:52.300 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:52.431 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:52.432 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:52.432 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:52.432 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:52.432 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:52.432 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T02:38:52.432 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-15T02:38:52.432 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T02:38:52.433 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:52.433 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:52.433 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:52.433 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-09-15T02:38:52.433 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:52.433 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 893 k 2024-09-15T02:38:52.434 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 2.6 M 2024-09-15T02:38:52.434 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:52.596 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 563 kB/s | 80 kB 00:00 2024-09-15T02:38:52.729 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-15T02:38:52.730 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:52.730 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.9 MB/s | 893 kB 00:00 2024-09-15T02:38:52.742 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:52.762 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:52.763 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:52.841 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:52.841 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:53.030 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:53.177 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T02:38:53.246 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:38:53.935 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:38:53.936 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T02:38:54.237 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T02:38:54.237 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:54.237 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:54.237 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T02:38:54.237 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:54.237 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:54.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-09-15T02:38:55.095 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:55.228 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:55.229 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 270 k 2024-09-15T02:38:55.230 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.3 M 2024-09-15T02:38:55.230 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:55.459 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T02:38:55.460 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:55.460 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-15T02:38:55.460 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:55.468 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:55.468 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:55.497 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:55.498 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:55.705 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:55.777 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:38:55.794 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:38:55.794 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:38:55.794 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T02:38:55.794 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:56.802 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:38:56.802 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:56.802 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:56.802 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:38:56.803 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:56.803 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:57.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-09-15T02:38:57.705 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:38:57.839 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:38:57.840 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:57.840 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-15T02:38:57.840 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:57.840 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:38:57.840 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T02:38:57.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:57.840 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:38:57.841 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:38:57.841 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:38:57.841 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:57.841 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 126 k 2024-09-15T02:38:57.841 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 448 k 2024-09-15T02:38:57.841 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:38:58.106 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 477 kB/s | 126 kB 00:00 2024-09-15T02:38:58.106 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:38:58.107 INFO:teuthology.orchestra.run.smithi137.stdout:Total 474 kB/s | 126 kB 00:00 2024-09-15T02:38:58.107 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:38:58.112 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:38:58.112 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:38:58.174 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:38:58.175 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:38:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:38:58.428 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:38:58.856 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:38:59.120 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:38:59.120 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:59.120 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:38:59.120 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:38:59.120 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:38:59.120 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:38:59.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-09-15T02:38:59.944 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:30 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:00.039 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:39:00.084 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:00.085 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:39:00.085 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:00.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-09-15T02:39:00.729 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:00.861 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:00.862 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:39:00.863 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:00.863 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 25 k 2024-09-15T02:39:00.863 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 110 k 2024-09-15T02:39:00.863 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:39:01.009 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 171 kB/s | 25 kB 00:00 2024-09-15T02:39:01.009 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:39:01.010 INFO:teuthology.orchestra.run.smithi137.stdout:Total 169 kB/s | 25 kB 00:00 2024-09-15T02:39:01.010 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:39:01.013 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:39:01.013 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:39:01.015 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:39:01.015 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:39:01.099 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:39:01.238 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:01.603 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:01.833 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:01.833 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:01.833 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:39:01.834 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:39:01.834 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:01.834 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:02.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-09-15T02:39:02.603 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:02.694 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:39:02.737 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:02.738 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:39:02.738 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:02.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-09-15T02:39:03.401 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:03.492 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:39:03.536 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:03.537 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:39:03.537 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:03.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-09-15T02:39:04.190 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:04.280 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:39:04.323 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:04.324 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:39:04.324 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:04.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-09-15T02:39:04.968 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:05.060 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:39:05.103 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:05.104 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:39:05.104 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:05.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-09-15T02:39:05.753 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:05.844 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:39:05.886 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:05.887 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:39:05.887 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:05.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-09-15T02:39:06.525 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:06.614 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:39:06.656 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:06.657 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-15T02:39:06.658 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:06.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-09-15T02:39:07.309 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:07.443 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:07.444 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:07.444 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-15T02:39:07.444 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:07.444 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:39:07.444 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T02:39:07.444 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:07.445 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:39:07.445 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:07.445 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:39:07.445 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:07.445 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 82 k 2024-09-15T02:39:07.445 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 222 k 2024-09-15T02:39:07.445 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:39:07.637 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 427 kB/s | 82 kB 00:00 2024-09-15T02:39:07.638 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:39:07.638 INFO:teuthology.orchestra.run.smithi137.stdout:Total 424 kB/s | 82 kB 00:00 2024-09-15T02:39:07.638 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:39:07.644 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:39:07.644 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:39:07.707 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:39:07.707 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:39:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:39:07.943 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:08.328 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:08.566 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:08.566 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:08.567 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:39:08.567 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:39:08.567 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:08.567 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:08.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-09-15T02:39:09.395 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:09.528 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:39:09.529 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:09.530 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:39:09.530 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:09.530 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.9 M 2024-09-15T02:39:09.530 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 12 M 2024-09-15T02:39:09.530 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:39:10.164 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 4.6 MB/s | 2.9 MB 00:00 2024-09-15T02:39:10.165 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:39:10.166 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.6 MB/s | 2.9 MB 00:00 2024-09-15T02:39:10.166 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:39:10.177 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:39:10.177 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:39:10.254 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:39:10.254 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:39:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:39:10.824 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:10.850 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:10.850 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:39:10.850 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T02:39:10.850 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:39:10.850 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:39:10.850 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:11.833 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:11.834 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:11.834 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:39:11.834 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:39:11.834 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:11.834 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:12.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-09-15T02:39:12.640 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 02:37:29 AM UTC. 2024-09-15T02:39:12.777 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-15T02:39:12.778 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:12.778 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-15T02:39:12.778 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:12.778 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-15T02:39:12.778 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 165 k 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 477 k 2024-09-15T02:39:12.779 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-15T02:39:12.993 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 774 kB/s | 165 kB 00:00 2024-09-15T02:39:12.993 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:39:12.994 INFO:teuthology.orchestra.run.smithi137.stdout:Total 768 kB/s | 165 kB 00:00 2024-09-15T02:39:12.994 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-15T02:39:13.002 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-15T02:39:13.002 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-15T02:39:13.065 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-15T02:39:13.066 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-15T02:39:13.221 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-15T02:39:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:13.765 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:39:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:14.017 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-15T02:39:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:39:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-15T02:39:14.017 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-15T02:39:14.271 DEBUG:teuthology.parallel:result is None 2024-09-15T02:39:14.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:39:14.424 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T02:39:14.444 INFO:teuthology.orchestra.run.smithi137.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T02:39:14.445 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T02:39:14.445 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T02:39:14.447 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T02:39:14.447 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-15T02:39:14.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:39:14.510 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T02:39:14.568 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-15T02:39:14.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:39:14.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:39:14.684 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T02:39:14.685 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-15T02:39:14.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:39:14.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:39:14.820 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T02:39:14.820 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-15T02:39:14.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:39:14.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:39:14.949 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T02:39:14.972 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:39:14.972 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T02:39:14.972 INFO:tasks.workunit:timeout=3h 2024-09-15T02:39:14.972 INFO:tasks.workunit:cleanup=True 2024-09-15T02:39:14.973 DEBUG:teuthology.orchestra.run.smithi137:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:39:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:39:15.006 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T02:39:15.006 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:39:15.061 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T02:39:15.062 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T02:39:15.118 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:39:15.173 INFO:tasks.workunit.client.0.smithi137.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T02:40:04.130 INFO:tasks.workunit.client.0.smithi137.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-15T02:40:04.130 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-15T02:40:04.130 INFO:tasks.workunit.client.0.smithi137.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr: git switch -c 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:Or undo this operation with: 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr: git switch - 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-15T02:40:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T02:40:04.132 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-15T02:40:04.132 INFO:tasks.workunit.client.0.smithi137.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:40:04.134 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T02:40:04.191 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-15T02:40:04.191 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T02:40:04.247 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-09-15T02:40:04.248 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-09-15T02:40:04.248 DEBUG:teuthology.orchestra.run.smithi137:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-09-15T02:40:04.326 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:40:04.331 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/mclock-config 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-15T02:40:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-15T02:40:04.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-09-15T02:40:04.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-09-15T02:40:04.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none ' 2024-09-15T02:40:04.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-15T02:40:04.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-09-15T02:40:04.341 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-09-15T02:40:04.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_profile_builtin_to_custom 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_profile_custom_to_builtin 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_profile_disallow_builtin_params_modify 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_profile_disallow_builtin_params_override 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_recovery_limit_adjustment_mclock' 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-15T02:40:04.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:40:04.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:40:04.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:40:04.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:40:04.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:40:04.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:40:04.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:40:04.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:40:04.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:40:04.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:04.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:40:04.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:40:04.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:04.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:40:04.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:40:04.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:40:04.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:40:04.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:04.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:04.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:40:04.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:40:04.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:40:04.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:40:04.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:40:04.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:04.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:04.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-09-15T02:40:04.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:40:04.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:40:04.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:40:04.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:40:04.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:40:04.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:40:04.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:40:04.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:40:04.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:40:04.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:04.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:40:04.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:40:04.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:04.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:40:04.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:40:04.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:40:04.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:40:04.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:04.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:04.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:40:04.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:40:04.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:40:04.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:40:04.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:40:04.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:04.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:04.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:40:04.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-15T02:40:04.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-15T02:40:04.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:40:04.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:04.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:04.731 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:04.731 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:04.731 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:04.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:04.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:40:04.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:40:04.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:40:04.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:40:04.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:40:04.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:40:04.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:40:04.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:40:04.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:40:04.966 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:40:04.966 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:04.966 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:04.966 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:04.966 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:40:04.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:40:04.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get fsid 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:05.047 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:05.048 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:05.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:40:05.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:40:05.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get mon_host 2024-09-15T02:40:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-15T02:40:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-15T02:40:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:40:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:40:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:40:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-15T02:40:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:40:05.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:40:05.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:05.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:05.949 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:05.949 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:05.949 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:05.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:05.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:40:05.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:40:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:05.975 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:40:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-15T02:40:05.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-15T02:40:05.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:40:05.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3f2ac09-51bd-43e6-a490-778cee797dc9 2024-09-15T02:40:05.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f3f2ac09-51bd-43e6-a490-778cee797dc9' 2024-09-15T02:40:05.978 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 f3f2ac09-51bd-43e6-a490-778cee797dc9 2024-09-15T02:40:05.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:40:05.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFSeZm+wrLOhAAoqi/b6tH9o3R7YuSM+heDg== 2024-09-15T02:40:05.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFSeZm+wrLOhAAoqi/b6tH9o3R7YuSM+heDg=="}' 2024-09-15T02:40:05.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3f2ac09-51bd-43e6-a490-778cee797dc9 -i td/mclock-config/0/new.json 2024-09-15T02:40:06.310 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:40:06.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-15T02:40:06.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAFSeZm+wrLOhAAoqi/b6tH9o3R7YuSM+heDg== --osd-uuid f3f2ac09-51bd-43e6-a490-778cee797dc9 2024-09-15T02:40:06.392 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:06.391+0000 7f8bc8622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:06.394 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:06.392+0000 7f8bc8622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:06.395 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:06.393+0000 7f8bc8622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:06.395 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:06.394+0000 7f8bc8622440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:40:06.396 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:06.394+0000 7f8bc8622440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:40:06.396 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:06.394+0000 7f8bc8622440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:40:06.473 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:06.471+0000 7f8bc8622440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-15T02:40:12.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-15T02:40:12.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:40:12.211 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:40:12.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:40:12.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:40:12.687 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:40:12.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:40:12.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-15T02:40:12.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:40:12.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:40:12.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:40:12.709 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:12.707+0000 7f9011324440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:12.711 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:12.709+0000 7f9011324440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:12.712 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:12.710+0000 7f9011324440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:13.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:40:13.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:40:13.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:40:13.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:40:13.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:40:13.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:13.049 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:40:13.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:40:13.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:13.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:13.225 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:13.224+0000 7f9011324440 -1 Falling back to public interface 2024-09-15T02:40:13.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:14.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:14.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:14.427 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:40:14.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:40:14.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:14.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:14.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:15.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:15.665+0000 7f9011324440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:40:15.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:15.781 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:40:15.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:15.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:40:15.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:15.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:16.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:17.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:17.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:17.300 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:40:17.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:40:17.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:17.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:17.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:18.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:18.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:18.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:40:18.669 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:40:18.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:18.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:19.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:20.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:20.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:20.061 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:40:20.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:40:20.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:20.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:20.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:21.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:21.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:21.433 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:40:21.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:40:21.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:21.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:21.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:22.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:22.823 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:40:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:40:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:23.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:24.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:24.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:24.292 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:40:24.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:40:24.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:24.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:24.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:25.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:25.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:25.650 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:40:25.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:40:25.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:25.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:26.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:27.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:27.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:27.015 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:40:27.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:40:27.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:27.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:27.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:28.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:28.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:28.830 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:40:28.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:40:28.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:28.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:29.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:30.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:30.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:30.637 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:40:30.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:40:30.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:30.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:31.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:32.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:32.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:32.140 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:40:32.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:40:32.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:32.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:32.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:33.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:33.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:33.800 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:40:33.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:40:33.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:33.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:34.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:35.511 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:40:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:40:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:36.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:37.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:37.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:37.590 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:40:37.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:40:37.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:37.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:38.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:39.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:39.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:39.777 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:40:39.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:40:39.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:39.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:40.151 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/28318,v1:127.0.0.1:6803/28318] [v2:127.0.0.1:6804/28318,v1:127.0.0.1:6805/28318] exists,up f3f2ac09-51bd-43e6-a490-778cee797dc9 2024-09-15T02:40:40.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:40:40.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:40:40.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:40:40.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-15T02:40:40.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:40.153 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:40.153 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:40.153 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:40.154 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:40.154 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:40.154 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:40.154 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:40.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:40.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:40:40.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-09-15T02:40:40.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-09-15T02:40:40.233 INFO:tasks.workunit.client.0.smithi137.stdout:osd_max_backfills: 1 2024-09-15T02:40:40.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-15T02:40:40.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-15T02:40:42.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-15T02:40:42.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:42.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:40:42.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-09-15T02:40:42.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:42.784 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:42.785 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:42.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:42.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok dump_recovery_reservations 2024-09-15T02:40:42.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-15T02:40:42.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-15T02:40:42.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-15T02:40:42.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:42.869 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:42.869 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:42.869 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:42.870 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:42.870 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:42.870 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:42.870 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:42.870 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:42.870 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok dump_recovery_reservations 2024-09-15T02:40:42.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-15T02:40:42.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-15T02:40:42.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-15T02:40:43.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-15T02:40:43.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-15T02:40:45.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-15T02:40:45.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:45.745 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:45.745 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:45.745 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:45.745 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:45.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:45.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:45.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:45.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:45.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:40:45.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-15T02:40:45.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-15T02:40:45.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-15T02:40:45.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:45.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok dump_recovery_reservations 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:45.919 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:45.920 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:45.920 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:45.920 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:45.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:45.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok dump_recovery_reservations 2024-09-15T02:40:46.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-15T02:40:46.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-15T02:40:46.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-09-15T02:40:46.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:40:46.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:40:46.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:40:46.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:40:46.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:40:46.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:40:46.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-09-15T02:40:46.669 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already down. 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:40:46.680 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:40:46.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:46.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-15T02:40:47.024 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/28318,v1:127.0.0.1:6803/28318] [v2:127.0.0.1:6804/28318,v1:127.0.0.1:6805/28318] exists f3f2ac09-51bd-43e6-a490-778cee797dc9 2024-09-15T02:40:47.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:40:47.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:40:47.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:40:47.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-09-15T02:40:47.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-09-15T02:40:47.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:40:47.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T02:40:47.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:40:47.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-09-15T02:40:47.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:47.032 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:40:47.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:40:47.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:40:47.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-09-15T02:40:47.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-09-15T02:40:47.034 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:40:47.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T02:40:47.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-09-15T02:40:47.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-09-15T02:40:47.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T02:40:47.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:40:47.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:40:47.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:40:47.048 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:47.047+0000 7f71c891f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:47.073 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:47.072+0000 7f71c891f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:47.074 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:47.073+0000 7f71c891f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:47.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T02:40:47.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:40:47.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:40:47.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:40:47.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:40:47.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:47.402 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:40:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:40:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:47.590 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:47.590+0000 7f71c891f440 -1 Falling back to public interface 2024-09-15T02:40:47.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:48.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:48.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:48.795 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:40:48.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:40:48.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:48.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:49.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:50.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:50.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:50.162 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:40:50.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:40:50.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:50.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:50.329 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:50.328+0000 7f71c891f440 -1 osd.0 6 log_to_monitors true 2024-09-15T02:40:50.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:51.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:51.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:51.569 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:40:51.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:40:51.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:51.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:40:51.916 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/29429,v1:127.0.0.1:6803/29429] [v2:127.0.0.1:6804/29429,v1:127.0.0.1:6805/29429] exists,up f3f2ac09-51bd-43e6-a490-778cee797dc9 2024-09-15T02:40:51.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:40:51.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:40:51.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:40:51.917 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-15T02:40:51.917 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:51.917 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:51.918 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:40:51.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-15T02:40:51.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-15T02:40:51.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-15T02:40:51.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:51.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok dump_recovery_reservations 2024-09-15T02:40:52.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-15T02:40:52.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-15T02:40:52.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-15T02:40:52.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-09-15T02:40:52.081 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:40:52.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok dump_recovery_reservations 2024-09-15T02:40:52.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-15T02:40:52.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:40:52.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:40:52.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:40:52.266 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:40:52.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:40:52.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:40:52.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:40:52.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:40:52.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:40:52.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:52.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:40:52.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:40:52.270 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:52.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:40:52.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:40:52.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:40:52.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:40:52.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:40:52.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:40:52.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:40:52.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:40:52.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:40:52.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:40:52.281 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:40:52.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:40:52.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:40:52.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:40:52.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:52.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:40:52.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:40:52.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:52.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:40:52.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:40:52.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:40:52.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:40:52.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:40:52.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:40:52.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:40:52.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-15T02:40:52.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-15T02:40:52.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:40:52.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:40:52.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:40:52.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:40:52.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:40:52.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:40:52.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:40:52.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:40:52.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:40:52.291 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:40:52.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:52.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:40:52.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:40:52.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:40:52.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:40:52.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:40:52.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:40:52.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:40:52.296 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.296 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:40:52.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:40:52.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:40:52.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:40:52.298 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:40:52.298 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.298 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:40:52.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:40:52.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:40:52.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:40:52.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:40:52.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-09-15T02:40:52.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-09-15T02:40:52.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-09-15T02:40:52.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-15T02:40:52.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:40:52.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:40:52.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:40:52.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-15T02:40:52.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:52.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:40:52.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:40:52.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:40:52.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:40:52.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:40:52.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:40:52.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get fsid 2024-09-15T02:40:52.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:40:52.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:40:52.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:40:52.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:40:52.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get mon_host 2024-09-15T02:40:53.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-09-15T02:40:53.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-15T02:40:53.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:40:53.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:40:53.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:40:53.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-15T02:40:53.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:40:53.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:40:53.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:40:53.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-15T02:40:53.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:40:53.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:40:53.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-15T02:40:53.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:40:53.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2c044fd-4588-405a-b3cf-7e3e69b7294d 2024-09-15T02:40:53.907 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 f2c044fd-4588-405a-b3cf-7e3e69b7294d 2024-09-15T02:40:53.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f2c044fd-4588-405a-b3cf-7e3e69b7294d' 2024-09-15T02:40:53.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:40:53.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1SeZm31+iNhAAj3D9eQyCPXnJCzIStlGi/g== 2024-09-15T02:40:53.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1SeZm31+iNhAAj3D9eQyCPXnJCzIStlGi/g=="}' 2024-09-15T02:40:53.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2c044fd-4588-405a-b3cf-7e3e69b7294d -i td/mclock-config/0/new.json 2024-09-15T02:40:54.239 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:40:54.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-15T02:40:54.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQA1SeZm31+iNhAAj3D9eQyCPXnJCzIStlGi/g== --osd-uuid f2c044fd-4588-405a-b3cf-7e3e69b7294d 2024-09-15T02:40:54.273 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:54.271+0000 7ff83ea67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:54.284 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:54.283+0000 7ff83ea67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:54.285 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:54.284+0000 7ff83ea67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:54.286 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:54.284+0000 7ff83ea67440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:40:54.286 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:54.284+0000 7ff83ea67440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:40:54.286 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:54.284+0000 7ff83ea67440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:40:54.324 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:40:54.323+0000 7ff83ea67440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-15T02:40:59.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-15T02:40:59.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:40:59.594 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:40:59.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:40:59.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:41:00.081 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:41:00.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:41:00.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-15T02:41:00.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:41:00.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:41:00.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:41:00.103 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:00.101+0000 7f95d351d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:00.105 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:00.103+0000 7f95d351d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:00.106 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:00.104+0000 7f95d351d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:00.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:41:00.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:41:00.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:41:00.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:41:00.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:41:00.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:00.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:41:00.428 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:41:00.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:00.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:00.629 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:00.627+0000 7f95d351d440 -1 Falling back to public interface 2024-09-15T02:41:00.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:01.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:01.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:01.777 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:41:01.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:41:01.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:01.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:02.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:03.062 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:03.060+0000 7f95d351d440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:41:03.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:03.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:03.135 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:41:03.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:41:03.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:03.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:03.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:04.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:04.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:04.606 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:41:04.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:41:04.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:04.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:05.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:06.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:06.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:06.042 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:41:06.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:41:06.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:06.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:06.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:07.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:07.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:07.415 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:41:07.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:41:07.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:07.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:07.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:08.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:08.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:08.879 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:41:08.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:41:08.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:08.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:09.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:10.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:10.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:10.268 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:41:10.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:41:10.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:10.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:10.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:11.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:11.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:11.692 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:41:11.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:41:11.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:11.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:12.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:13.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:13.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:13.050 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:41:13.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:41:13.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:13.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:13.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:14.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:14.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:14.435 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:41:14.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:41:14.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:14.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:15.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:16.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:16.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:16.364 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:41:16.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:41:16.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:16.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:16.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:17.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:17.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:17.730 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:41:17.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:41:17.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:17.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:18.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:19.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:19.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:19.629 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:41:19.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:41:19.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:19.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:20.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:21.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:21.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:21.330 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:41:21.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:41:21.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:21.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:22.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:23.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:23.188 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:41:23.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:23.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:41:23.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:23.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:23.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:24.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:24.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:24.917 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:41:24.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:41:24.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:24.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:25.269 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/31131,v1:127.0.0.1:6803/31131] [v2:127.0.0.1:6804/31131,v1:127.0.0.1:6805/31131] exists,up f2c044fd-4588-405a-b3cf-7e3e69b7294d 2024-09-15T02:41:25.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:41:25.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:41:25.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:41:25.270 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-09-15T02:41:25.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-09-15T02:41:25.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:41:25.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-15T02:41:25.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_profile 2024-09-15T02:41:25.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-09-15T02:41:25.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-09-15T02:41:25.719 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-09-15T02:41:25.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-09-15T02:41:25.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-09-15T02:41:25.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-15T02:41:25.814 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:41:25.814 INFO:tasks.workunit.client.0.smithi137.stdout: "success": "" 2024-09-15T02:41:25.814 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-15T02:41:25.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_profile 2024-09-15T02:41:25.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-09-15T02:41:25.901 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-09-15T02:41:25.901 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-09-15T02:41:25.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-09-15T02:41:25.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:25.903 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:41:25.904 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-15T02:41:25.904 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:41:25.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-09-15T02:41:25.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-09-15T02:41:25.981 INFO:tasks.workunit.client.0.smithi137.stdout:client_res = 0.300000 2024-09-15T02:41:25.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-09-15T02:41:25.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-09-15T02:41:25.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-09-15T02:41:25.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-09-15T02:41:25.982 INFO:tasks.workunit.client.0.smithi137.stdout:client_res_new = .400000 2024-09-15T02:41:25.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-15T02:41:26.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-15T02:41:26.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-09-15T02:41:26.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-15T02:41:26.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-09-15T02:41:26.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:26.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:41:26.747 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-15T02:41:26.747 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:41:26.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-09-15T02:41:26.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-15T02:41:26.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:41:26.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:41:26.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:41:26.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:41:26.932 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:41:26.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:41:26.933 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:41:26.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:41:26.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:41:26.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:41:26.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:41:26.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:41:26.935 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:41:26.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:41:26.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:41:26.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:41:26.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:26.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:26.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:41:26.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:41:26.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:41:26.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:41:26.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:41:26.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:41:26.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:41:26.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:41:26.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:41:26.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:41:26.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:41:26.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:41:26.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:41:26.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:41:26.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:41:26.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:41:26.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:41:26.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:41:26.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:26.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:26.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:41:26.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:41:26.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:41:26.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:41:26.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:41:26.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:41:26.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:41:26.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:41:26.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:41:26.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:41:26.956 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:41:26.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:41:26.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:41:26.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:41:26.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:41:26.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:41:26.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:41:26.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:26.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:26.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:41:26.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:41:26.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:41:26.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:41:26.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:41:26.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:26.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:26.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:41:26.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:41:26.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:41:26.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:41:26.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:41:26.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-09-15T02:41:26.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:41:26.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:41:26.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:41:26.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:41:26.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:41:26.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:41:26.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:41:26.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:41:26.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:41:26.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:41:26.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:41:26.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:41:26.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:41:26.972 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:41:26.972 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:26.972 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:26.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:41:26.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:41:26.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:41:26.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:41:26.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:41:26.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:26.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:26.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:41:26.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:41:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:41:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:41:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:41:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-09-15T02:41:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-15T02:41:26.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:41:26.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:41:26.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:41:26.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-15T02:41:26.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:41:27.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:41:27.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:41:27.581 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:41:27.581 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:27.582 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:41:27.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:41:27.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get fsid 2024-09-15T02:41:27.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:41:27.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:41:27.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:41:27.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:41:27.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:41:27.745 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:41:27.745 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get mon_host 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:41:27.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-15T02:41:27.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:41:28.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:41:28.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:41:28.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:41:28.569 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:41:28.569 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:28.569 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:28.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:41:28.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:41:28.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:41:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-15T02:41:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-15T02:41:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:41:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:41:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:41:28.595 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:41:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:41:28.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:41:28.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:41:28.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:41:28.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-15T02:41:28.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-15T02:41:28.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:41:28.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b09e3253-c0a9-4cfd-921f-572b84031648 2024-09-15T02:41:28.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b09e3253-c0a9-4cfd-921f-572b84031648' 2024-09-15T02:41:28.599 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 b09e3253-c0a9-4cfd-921f-572b84031648 2024-09-15T02:41:28.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:41:28.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYSeZmhkQ3JBAAm4pRh1Bo7G3+cx9p8c2OKQ== 2024-09-15T02:41:28.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYSeZmhkQ3JBAAm4pRh1Bo7G3+cx9p8c2OKQ=="}' 2024-09-15T02:41:28.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b09e3253-c0a9-4cfd-921f-572b84031648 -i td/mclock-config/0/new.json 2024-09-15T02:41:28.913 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:41:28.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-15T02:41:28.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBYSeZmhkQ3JBAAm4pRh1Bo7G3+cx9p8c2OKQ== --osd-uuid b09e3253-c0a9-4cfd-921f-572b84031648 2024-09-15T02:41:28.947 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:28.946+0000 7f5b55e15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:28.959 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:28.958+0000 7f5b55e15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:28.960 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:28.959+0000 7f5b55e15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:28.961 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:28.960+0000 7f5b55e15440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:41:28.961 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:28.960+0000 7f5b55e15440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:41:28.961 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:28.960+0000 7f5b55e15440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:41:28.999 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:28.998+0000 7f5b55e15440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-15T02:41:34.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-15T02:41:34.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:41:34.320 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:41:34.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:41:34.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:41:34.797 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:41:34.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:41:34.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-15T02:41:34.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:41:34.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:41:34.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:41:34.818 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:34.818+0000 7f8eaae2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:34.820 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:34.820+0000 7f8eaae2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:34.821 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:34.821+0000 7f8eaae2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:35.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:41:35.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:41:35.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:41:35.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:41:35.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:41:35.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:35.149 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:41:35.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:41:35.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:35.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:35.343 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:35.343+0000 7f8eaae2b440 -1 Falling back to public interface 2024-09-15T02:41:35.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:36.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:36.504 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:41:36.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:36.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:41:36.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:36.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:36.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:37.808 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:41:37.807+0000 7f8eaae2b440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:41:37.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:37.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:37.859 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:41:37.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:41:37.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:37.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:38.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:39.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:39.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:39.344 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:41:39.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:41:39.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:39.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:39.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:40.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:40.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:40.773 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:41:40.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:41:40.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:40.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:41.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:42.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:42.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:42.180 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:41:42.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:41:42.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:42.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:42.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:43.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:43.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:43.612 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:41:43.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:41:43.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:43.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:43.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:44.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:44.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:44.998 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:41:44.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:41:44.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:44.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:45.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:46.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:46.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:46.481 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:41:46.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:41:46.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:46.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:46.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:47.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:47.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:47.838 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:41:47.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:41:47.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:47.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:48.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:49.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:49.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:49.217 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:41:49.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:41:49.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:49.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:49.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:50.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:50.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:50.937 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:41:50.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:41:50.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:50.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:51.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:52.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:52.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:52.589 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:41:52.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:41:52.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:52.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:53.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:54.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:54.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:54.338 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:41:54.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:41:54.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:54.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:55.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:56.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:56.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:56.480 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:41:56.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:41:56.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:56.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:57.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:58.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:58.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:58.208 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:41:58.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:41:58.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:58.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:41:58.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:59.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:59.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:59.720 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:41:59.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:41:59.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:59.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:00.150 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/33283,v1:127.0.0.1:6803/33283] [v2:127.0.0.1:6804/33283,v1:127.0.0.1:6805/33283] exists,up b09e3253-c0a9-4cfd-921f-572b84031648 2024-09-15T02:42:00.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:42:00.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:42:00.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:42:00.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-09-15T02:42:00.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-09-15T02:42:00.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-09-15T02:42:00.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-15T02:42:00.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_profile 2024-09-15T02:42:00.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-09-15T02:42:00.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-09-15T02:42:00.644 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-09-15T02:42:00.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-09-15T02:42:00.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-15T02:42:00.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-15T02:42:00.741 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:42:00.741 INFO:tasks.workunit.client.0.smithi137.stdout: "success": "" 2024-09-15T02:42:00.742 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:42:00.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-15T02:42:00.751 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-15T02:42:00.751 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:00.751 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:00.752 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:00.752 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:00.752 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:00.752 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:00.752 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-15T02:42:00.752 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_profile 2024-09-15T02:42:00.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-09-15T02:42:00.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-09-15T02:42:00.831 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-09-15T02:42:00.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-09-15T02:42:00.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-09-15T02:42:00.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-09-15T02:42:00.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-15T02:42:00.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-09-15T02:42:00.832 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-15T02:42:00.832 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:00.832 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:00.833 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:00.833 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:00.833 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:00.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:00.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-15T02:42:00.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:42:00.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-09-15T02:42:00.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-09-15T02:42:00.909 INFO:tasks.workunit.client.0.smithi137.stdout:Original client_res for osd.0 = 0.300000 2024-09-15T02:42:00.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-09-15T02:42:00.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-09-15T02:42:00.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-09-15T02:42:00.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-09-15T02:42:00.911 INFO:tasks.workunit.client.0.smithi137.stdout:client_res_new = .400000 2024-09-15T02:42:00.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-09-15T02:42:01.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-15T02:42:01.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-15T02:42:01.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-15T02:42:01.752 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-09-15T02:42:01.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:01.754 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:01.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-15T02:42:01.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:42:01.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-09-15T02:42:01.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-15T02:42:01.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-09-15T02:42:01.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-15T02:42:01.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-09-15T02:42:01.962 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:42:01.962 INFO:tasks.workunit.client.0.smithi137.stdout: "success": "" 2024-09-15T02:42:01.962 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:42:01.972 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-15T02:42:01.972 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-15T02:42:01.973 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_profile 2024-09-15T02:42:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-09-15T02:42:02.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-09-15T02:42:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-15T02:42:02.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-15T02:42:02.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-15T02:42:02.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-15T02:42:02.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-09-15T02:42:02.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-15T02:42:02.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-15T02:42:02.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-09-15T02:42:02.425 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-15T02:42:02.425 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:02.425 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:02.425 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:02.426 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:02.426 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:02.426 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:02.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-15T02:42:02.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:42:02.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-09-15T02:42:02.504 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-15T02:42:02.504 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-09-15T02:42:02.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-15T02:42:02.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-09-15T02:42:03.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-09-15T02:42:08.004 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-15T02:42:08.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-09-15T02:42:08.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-09-15T02:42:08.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-09-15T02:42:08.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-15T02:42:08.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-15T02:42:08.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-09-15T02:42:08.390 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-15T02:42:08.390 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:08.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:08.391 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:08.391 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.391 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:08.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-15T02:42:08.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:42:08.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-09-15T02:42:08.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-09-15T02:42:08.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-09-15T02:42:08.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-15T02:42:08.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-09-15T02:42:08.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:08.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:08.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:08.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:08.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:08.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:08.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:08.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:08.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:08.577 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:08.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:08.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:08.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:08.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:08.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:08.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:08.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:08.581 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:08.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:08.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:08.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:08.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:08.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:08.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:08.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-15T02:42:08.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:08.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:08.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:08.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:08.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:08.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:08.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:08.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:08.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:08.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:08.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:08.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:08.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:08.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:08.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:08.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:08.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:08.595 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:08.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:08.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:08.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:08.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:08.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:08.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:08.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:08.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:08.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:08.601 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:08.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:08.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:08.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:08.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:08.603 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:08.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:08.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:08.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:08.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:08.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:08.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:08.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:08.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:08.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:08.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:42:08.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:42:08.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:42:08.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:42:08.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:08.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:08.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:08.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:08.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:08.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:08.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:08.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:08.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:08.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:08.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:08.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:08.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:08.617 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:08.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:08.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:08.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:08.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:08.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:08.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:08.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:08.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:42:08.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:42:08.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-15T02:42:08.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-15T02:42:08.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:42:08.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:42:08.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:42:08.927 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:42:08.927 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:08.927 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:08.928 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:08.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:42:09.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:42:09.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:42:09.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get fsid 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:09.253 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:09.254 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:09.254 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:42:09.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:42:09.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get mon_host 2024-09-15T02:42:09.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-09-15T02:42:09.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-15T02:42:09.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:42:09.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:42:09.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:42:09.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-15T02:42:09.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:10.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:42:10.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:42:10.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-15T02:42:10.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-15T02:42:10.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:42:10.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:42:10.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-15T02:42:10.190 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:42:10.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-15T02:42:10.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:42:10.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c066aca7-334e-4656-91da-52e32c620425 2024-09-15T02:42:10.193 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 c066aca7-334e-4656-91da-52e32c620425 2024-09-15T02:42:10.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c066aca7-334e-4656-91da-52e32c620425' 2024-09-15T02:42:10.194 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:42:10.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCSeZm/p0JDBAANKSEf5D/wrZLPxF8ymjdAg== 2024-09-15T02:42:10.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCSeZm/p0JDBAANKSEf5D/wrZLPxF8ymjdAg=="}' 2024-09-15T02:42:10.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c066aca7-334e-4656-91da-52e32c620425 -i td/mclock-config/0/new.json 2024-09-15T02:42:10.504 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:42:10.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-15T02:42:10.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCCSeZm/p0JDBAANKSEf5D/wrZLPxF8ymjdAg== --osd-uuid c066aca7-334e-4656-91da-52e32c620425 2024-09-15T02:42:10.538 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:10.536+0000 7f7d44d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:10.552 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:10.550+0000 7f7d44d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:10.553 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:10.551+0000 7f7d44d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:10.553 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:10.552+0000 7f7d44d21440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:42:10.553 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:10.552+0000 7f7d44d21440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:42:10.553 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:10.552+0000 7f7d44d21440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:42:10.592 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:10.590+0000 7f7d44d21440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-15T02:42:16.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-15T02:42:16.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:42:16.213 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:42:16.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:42:16.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:42:16.690 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:42:16.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:42:16.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-15T02:42:16.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:42:16.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:42:16.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:42:16.712 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:16.710+0000 7f55fec49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:16.713 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:16.712+0000 7f55fec49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:16.714 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:16.713+0000 7f55fec49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:17.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:42:17.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:42:17.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:42:17.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:42:17.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:42:17.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:17.039 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:42:17.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:42:17.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:17.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:17.235 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:17.234+0000 7f55fec49440 -1 Falling back to public interface 2024-09-15T02:42:17.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:18.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:18.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:18.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:42:18.389 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:42:18.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:18.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:18.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:19.719 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:19.717+0000 7f55fec49440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:42:19.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:19.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:19.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:42:19.744 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:42:19.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:19.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:20.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:21.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:21.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:21.104 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:42:21.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:42:21.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:21.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:21.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:22.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:22.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:22.583 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:42:22.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:42:22.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:22.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:22.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:23.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:23.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:23.956 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:42:23.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:42:23.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:23.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:24.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:25.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:25.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:25.404 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:42:25.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:42:25.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:25.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:25.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:26.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:26.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:26.791 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:42:26.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:42:26.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:26.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:27.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:28.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:28.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:28.191 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:42:28.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:42:28.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:28.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:28.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:29.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:29.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:29.552 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:42:29.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:42:29.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:29.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:29.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:30.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:30.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:30.954 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:42:30.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:42:30.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:30.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:31.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:32.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:32.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:32.391 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:42:32.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:42:32.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:32.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:33.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:34.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:34.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:34.192 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:42:34.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:42:34.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:34.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:34.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:35.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:35.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:35.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:42:35.903 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:42:35.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:35.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:36.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:37.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:37.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:37.769 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:42:37.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:42:37.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:37.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:38.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:39.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:39.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:39.690 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:42:39.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:42:39.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:39.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:40.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:41.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:41.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:41.156 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:42:41.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:42:41.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:41.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:41.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:42.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:42.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:42.693 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:42:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:42:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:42.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:43.082 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/35556,v1:127.0.0.1:6803/35556] [v2:127.0.0.1:6804/35556,v1:127.0.0.1:6805/35556] exists,up c066aca7-334e-4656-91da-52e32c620425 2024-09-15T02:42:43.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:42:43.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:42:43.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:42:43.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-09-15T02:42:43.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-09-15T02:42:43.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-09-15T02:42:43.485 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-09-15T02:42:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-15T02:42:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:43.486 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:43.486 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:43.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-15T02:42:43.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_profile 2024-09-15T02:42:43.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-15T02:42:43.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-09-15T02:42:43.566 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-15T02:42:43.567 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:43.568 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:43.568 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:43.568 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:43.568 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:43.568 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:43.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-15T02:42:43.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-15T02:42:43.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-09-15T02:42:43.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-09-15T02:42:43.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:43.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-09-15T02:42:43.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-09-15T02:42:44.085 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-15T02:42:44.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-15T02:42:44.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-15T02:42:44.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-15T02:42:44.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-15T02:42:46.116 INFO:tasks.workunit.client.0.smithi137.stdout:Check configuration values - Attempt#: 0 2024-09-15T02:42:46.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-15T02:42:46.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-15T02:42:46.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-15T02:42:46.513 INFO:tasks.workunit.client.0.smithi137.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-15T02:42:46.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-15T02:42:46.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-09-15T02:42:46.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:46.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-15T02:42:46.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-15T02:42:46.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-09-15T02:42:46.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-15T02:42:46.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-15T02:42:46.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-09-15T02:42:46.864 INFO:tasks.workunit.client.0.smithi137.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-15T02:42:46.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-15T02:42:46.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-09-15T02:42:46.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:46.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-09-15T02:42:46.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:46.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-15T02:42:46.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-15T02:42:46.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-15T02:42:46.868 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-15T02:42:46.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-15T02:42:46.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-09-15T02:42:46.946 INFO:tasks.workunit.client.0.smithi137.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-15T02:42:46.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-15T02:42:46.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-09-15T02:42:46.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:46.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-09-15T02:42:46.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:46.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-15T02:42:46.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-15T02:42:46.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-15T02:42:46.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:46.951 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:46.952 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:46.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-15T02:42:46.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:42:47.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-09-15T02:42:47.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-09-15T02:42:47.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:47.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-09-15T02:42:47.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-15T02:42:47.564 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-15T02:42:47.565 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-15T02:42:47.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-15T02:42:47.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-15T02:42:47.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-15T02:42:49.568 INFO:tasks.workunit.client.0.smithi137.stdout:Check configuration values - Attempt#: 0 2024-09-15T02:42:49.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-15T02:42:49.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-15T02:42:49.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-15T02:42:49.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-15T02:42:49.958 INFO:tasks.workunit.client.0.smithi137.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-15T02:42:49.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-09-15T02:42:49.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:49.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-15T02:42:49.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-15T02:42:49.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-09-15T02:42:49.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-15T02:42:49.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-15T02:42:50.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-09-15T02:42:50.315 INFO:tasks.workunit.client.0.smithi137.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-15T02:42:50.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-15T02:42:50.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-09-15T02:42:50.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:50.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-09-15T02:42:50.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:50.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-15T02:42:50.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-15T02:42:50.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:42:50.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-09-15T02:42:50.397 INFO:tasks.workunit.client.0.smithi137.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-15T02:42:50.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-15T02:42:50.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-09-15T02:42:50.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:50.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-09-15T02:42:50.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-15T02:42:50.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-15T02:42:50.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:50.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:50.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:50.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:50.506 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:50.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:50.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:50.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:50.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:50.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:50.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:50.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:50.510 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:50.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:50.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:50.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:50.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.517 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-15T02:42:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:50.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:50.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:50.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:50.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:50.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:50.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:50.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:50.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:50.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:50.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:50.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:50.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:50.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:50.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:50.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:50.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:50.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:50.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:50.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:50.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:50.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:50.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:50.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:50.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:50.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:50.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:50.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:50.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:50.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:50.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:50.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:50.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:50.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:50.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:50.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:50.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:50.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:50.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:50.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.536 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:50.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:50.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:50.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:42:50.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:42:50.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:42:50.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:42:50.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:42:50.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:42:50.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:42:50.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-09-15T02:42:50.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-09-15T02:42:50.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:42:50.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:42:50.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:42:50.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:42:50.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:42:50.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:42:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:42:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:42:50.543 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:42:50.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:42:50.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:42:50.545 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:42:50.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:42:50.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:42:50.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:42:50.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:42:50.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:42:50.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:42:50.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:42:50.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:42:50.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:42:50.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-15T02:42:50.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-15T02:42:50.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:42:50.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:42:50.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:42:50.947 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:42:50.947 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:50.947 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:50.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:50.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:42:51.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:51.183 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:51.184 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:42:51.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:42:51.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get fsid 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:51.259 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:51.260 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:51.260 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:42:51.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:42:51.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get mon_host 2024-09-15T02:42:51.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-09-15T02:42:51.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-15T02:42:51.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:42:51.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:42:51.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:42:51.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-15T02:42:51.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:42:52.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:42:52.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:42:52.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:42:52.184 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:42:52.184 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:52.184 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:52.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:52.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:42:52.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:42:52.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-15T02:42:52.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:42:52.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-15T02:42:52.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17a1a9dd-19d9-49e4-9784-6a571ae67253 2024-09-15T02:42:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 17a1a9dd-19d9-49e4-9784-6a571ae67253' 2024-09-15T02:42:52.213 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 17a1a9dd-19d9-49e4-9784-6a571ae67253 2024-09-15T02:42:52.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:42:52.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsSeZmg/E6DRAA6RnatOf8NdFDOu3UfO5ntw== 2024-09-15T02:42:52.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsSeZmg/E6DRAA6RnatOf8NdFDOu3UfO5ntw=="}' 2024-09-15T02:42:52.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17a1a9dd-19d9-49e4-9784-6a571ae67253 -i td/mclock-config/0/new.json 2024-09-15T02:42:52.486 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:42:52.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-15T02:42:52.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCsSeZmg/E6DRAA6RnatOf8NdFDOu3UfO5ntw== --osd-uuid 17a1a9dd-19d9-49e4-9784-6a571ae67253 2024-09-15T02:42:52.520 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:52.518+0000 7f2ebf814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:52.532 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:52.530+0000 7f2ebf814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:52.533 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:52.531+0000 7f2ebf814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:52.533 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:52.532+0000 7f2ebf814440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:42:52.533 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:52.532+0000 7f2ebf814440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:42:52.534 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:52.532+0000 7f2ebf814440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:42:52.563 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:52.562+0000 7f2ebf814440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-15T02:42:57.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-15T02:42:57.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:42:57.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:42:57.930 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:42:57.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:42:58.404 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:42:58.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:42:58.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-15T02:42:58.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:42:58.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:42:58.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:42:58.426 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:58.424+0000 7f0fb5a12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:58.427 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:58.426+0000 7f0fb5a12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:58.428 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:58.427+0000 7f0fb5a12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:58.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:42:58.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:42:58.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:42:58.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:42:58.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:42:58.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:58.770 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:42:58.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:42:58.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:58.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:42:58.942 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:42:58.941+0000 7f0fb5a12440 -1 Falling back to public interface 2024-09-15T02:42:59.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:00.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:00.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:00.121 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:43:00.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:43:00.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:00.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:00.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:01.433 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:01.431+0000 7f0fb5a12440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:43:01.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:01.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:01.477 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:43:01.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:43:01.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:01.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:02.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:03.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:03.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:03.010 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:43:03.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:43:03.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:03.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:03.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:04.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:04.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:04.424 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:43:04.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:43:04.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:04.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:04.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:05.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:05.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:05.783 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:43:05.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:43:05.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:05.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:06.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:07.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:07.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:07.235 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:43:07.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:43:07.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:07.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:07.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:08.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:08.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:08.632 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:43:08.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:43:08.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:08.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:09.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:10.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:10.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:10.107 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:43:10.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:43:10.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:10.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:10.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:11.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:11.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:11.510 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:43:11.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:43:11.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:11.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:11.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:12.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:12.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:12.891 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:43:12.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:43:12.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:12.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:13.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:14.636 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:43:14.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:43:14.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:14.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:15.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:16.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:16.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:16.125 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:43:16.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:43:16.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:16.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:17.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:18.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:18.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:18.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:43:18.186 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:43:18.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:18.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:18.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:19.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:19.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:19.938 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:43:19.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:43:19.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:19.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:20.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:21.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:21.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:21.708 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:43:21.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:43:21.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:21.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:22.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:23.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:23.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:23.767 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:43:23.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:43:23.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:23.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:24.120 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37849,v1:127.0.0.1:6803/37849] [v2:127.0.0.1:6804/37849,v1:127.0.0.1:6805/37849] exists,up 17a1a9dd-19d9-49e4-9784-6a571ae67253 2024-09-15T02:43:24.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:43:24.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:43:24.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:43:24.121 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-09-15T02:43:24.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-09-15T02:43:24.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-15T02:43:24.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_profile 2024-09-15T02:43:24.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-15T02:43:24.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:43:24.589 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:43:24.590 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:24.590 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:24.590 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:24.590 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:43:24.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-15T02:43:24.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-15T02:43:24.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-09-15T02:43:24.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-09-15T02:43:24.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:24.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-09-15T02:43:24.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-09-15T02:43:24.768 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:43:24.768 INFO:tasks.workunit.client.0.smithi137.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-09-15T02:43:24.768 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:43:24.777 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-15T02:43:24.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-15T02:43:24.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-15T02:43:24.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-15T02:43:24.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-15T02:43:26.781 INFO:tasks.workunit.client.0.smithi137.stdout:Check configuration values - Attempt#: 0 2024-09-15T02:43:26.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-15T02:43:26.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-15T02:43:27.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-15T02:43:27.139 INFO:tasks.workunit.client.0.smithi137.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-15T02:43:27.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-15T02:43:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-09-15T02:43:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-15T02:43:27.486 INFO:tasks.workunit.client.0.smithi137.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-15T02:43:27.487 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-09-15T02:43:27.487 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:27.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-09-15T02:43:27.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:27.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-15T02:43:27.493 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-15T02:43:27.493 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-15T02:43:27.493 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-15T02:43:27.493 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:43:27.493 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:43:27.493 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:27.493 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:27.494 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:27.494 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:43:27.494 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-15T02:43:27.494 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-15T02:43:27.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-09-15T02:43:27.568 INFO:tasks.workunit.client.0.smithi137.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-15T02:43:27.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-15T02:43:27.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-09-15T02:43:27.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:27.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-09-15T02:43:27.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:27.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-15T02:43:27.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-15T02:43:27.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-15T02:43:27.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-15T02:43:27.572 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-15T02:43:27.572 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-15T02:43:27.572 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-15T02:43:27.572 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:43:27.572 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:43:27.573 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:27.573 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:27.573 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:27.573 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:43:27.573 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-15T02:43:27.573 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:43:27.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-09-15T02:43:27.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-09-15T02:43:27.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:27.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-09-15T02:43:27.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-09-15T02:43:27.761 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:43:27.762 INFO:tasks.workunit.client.0.smithi137.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-09-15T02:43:27.762 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:43:27.771 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-15T02:43:27.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-15T02:43:27.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-15T02:43:27.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-15T02:43:27.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-15T02:43:29.774 INFO:tasks.workunit.client.0.smithi137.stdout:Check configuration values - Attempt#: 0 2024-09-15T02:43:29.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-15T02:43:29.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-15T02:43:30.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-15T02:43:30.126 INFO:tasks.workunit.client.0.smithi137.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-15T02:43:30.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-15T02:43:30.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-09-15T02:43:30.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:30.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-15T02:43:30.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-15T02:43:30.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-09-15T02:43:30.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-15T02:43:30.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-15T02:43:30.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-09-15T02:43:30.472 INFO:tasks.workunit.client.0.smithi137.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-15T02:43:30.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-15T02:43:30.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-09-15T02:43:30.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:30.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-09-15T02:43:30.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:30.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-15T02:43:30.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-15T02:43:30.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-15T02:43:30.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-15T02:43:30.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:43:30.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:43:30.477 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:30.477 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:30.477 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:30.477 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:43:30.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-15T02:43:30.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-15T02:43:30.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-09-15T02:43:30.554 INFO:tasks.workunit.client.0.smithi137.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-15T02:43:30.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-15T02:43:30.555 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-09-15T02:43:30.555 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:30.556 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-09-15T02:43:30.556 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-15T02:43:30.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-15T02:43:30.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-15T02:43:30.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:43:30.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:43:30.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:43:30.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:43:30.664 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:43:30.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:43:30.665 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:43:30.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:43:30.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:43:30.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:43:30.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:43:30.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:43:30.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:43:30.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:43:30.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:43:30.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:43:30.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:30.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:30.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:43:30.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:43:30.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:43:30.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-15T02:43:30.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:43:30.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:43:30.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:43:30.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:43:30.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:43:30.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:43:30.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:43:30.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:43:30.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:43:30.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:43:30.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:43:30.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:43:30.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:43:30.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:43:30.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:43:30.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:43:30.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:43:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:43:30.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:43:30.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:43:30.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:43:30.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:30.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:30.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:43:30.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:43:30.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:43:30.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:43:30.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:43:30.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:43:30.688 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:43:30.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:43:30.688 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:43:30.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:43:30.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:43:30.690 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:43:30.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:43:30.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:43:30.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:43:30.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:43:30.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:43:30.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:43:30.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:30.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:30.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:43:30.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:43:30.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:43:30.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:43:30.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:43:30.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:30.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:30.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-09-15T02:43:30.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:43:30.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:43:30.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:43:30.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:43:30.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:43:30.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:43:30.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:43:30.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:43:30.703 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:43:30.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:43:30.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:43:30.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:30.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:43:30.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:43:30.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:43:30.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:43:30.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:30.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:30.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:43:30.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:43:30.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:43:30.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-15T02:43:30.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:43:30.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:30.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:30.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27209 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-15T02:43:30.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-15T02:43:31.073 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:43:31.073 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:43:31.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:43:31.074 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:43:31.074 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:31.074 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:31.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:43:31.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:43:31.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:43:31.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:43:31.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:43:31.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:43:31.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:43:31.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:43:31.318 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:43:31.318 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:43:31.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:43:31.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:31.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:31.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:31.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:43:31.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:43:31.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get fsid 2024-09-15T02:43:31.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:31.397 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-mon.a.asok 2024-09-15T02:43:31.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:43:31.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27209/ceph-mon.a.asok config get mon_host 2024-09-15T02:43:31.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-15T02:43:31.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-15T02:43:31.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:43:31.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:43:31.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:43:31.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-15T02:43:31.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:43:32.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:43:32.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:43:32.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:43:32.295 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:43:32.295 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:32.295 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:32.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:43:32.296 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:43:32.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:43:32.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:43:32.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:43:32.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:43:32.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-15T02:43:32.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-15T02:43:32.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:43:32.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9929830f-c5ed-44e5-8cfd-4f55651f0ef4 2024-09-15T02:43:32.323 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 9929830f-c5ed-44e5-8cfd-4f55651f0ef4 2024-09-15T02:43:32.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9929830f-c5ed-44e5-8cfd-4f55651f0ef4' 2024-09-15T02:43:32.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:43:32.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUSeZmzGfYExAA8IcZU3FLSdo6sR3iFLM/gQ== 2024-09-15T02:43:32.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUSeZmzGfYExAA8IcZU3FLSdo6sR3iFLM/gQ=="}' 2024-09-15T02:43:32.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9929830f-c5ed-44e5-8cfd-4f55651f0ef4 -i td/mclock-config/0/new.json 2024-09-15T02:43:32.629 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:43:32.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-15T02:43:32.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDUSeZmzGfYExAA8IcZU3FLSdo6sR3iFLM/gQ== --osd-uuid 9929830f-c5ed-44e5-8cfd-4f55651f0ef4 2024-09-15T02:43:32.663 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:32.663+0000 7f3f4bf2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:32.676 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:32.675+0000 7f3f4bf2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:32.677 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:32.676+0000 7f3f4bf2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:32.677 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:32.677+0000 7f3f4bf2b440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:43:32.677 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:32.677+0000 7f3f4bf2b440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:43:32.677 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:32.677+0000 7f3f4bf2b440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-15T02:43:32.716 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:32.715+0000 7f3f4bf2b440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-15T02:43:38.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-15T02:43:38.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:43:38.111 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:43:38.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:43:38.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:43:38.591 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:43:38.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:43:38.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eaf92dae-40a6-4c6d-8c6f-74e4eccb5117 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.27209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-15T02:43:38.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:43:38.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:43:38.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:43:38.612 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:38.612+0000 7fed69414440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:38.614 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:38.614+0000 7fed69414440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:38.615 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:38.615+0000 7fed69414440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:38.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:43:38.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:43:38.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:43:38.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:43:38.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:43:38.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:38.942 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:43:38.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:43:38.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:38.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:39.130 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:39.130+0000 7fed69414440 -1 Falling back to public interface 2024-09-15T02:43:39.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:40.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:40.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:40.290 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:43:40.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:43:40.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:40.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:40.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:41.608 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:43:41.608+0000 7fed69414440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:43:41.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:41.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:41.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:43:41.650 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:43:41.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:41.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:42.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:43.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:43.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:43.188 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:43:43.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:43:43.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:43.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:43.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:44.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:44.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:44.549 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:43:44.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:43:44.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:44.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:44.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:45.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:46.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:46.018 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:43:46.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:43:46.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:46.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:46.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:47.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:47.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:47.385 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:43:47.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:43:47.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:47.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:47.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:48.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:48.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:48.744 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:43:48.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:43:48.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:48.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:49.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:50.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:50.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:50.105 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:43:50.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:43:50.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:50.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:50.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:51.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:51.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:51.462 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:43:51.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:43:51.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:51.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:51.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:52.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:52.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:52.831 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:43:52.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:43:52.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:52.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:53.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:54.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:54.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:54.275 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:43:54.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:43:54.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:54.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:54.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:55.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:55.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:55.853 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:43:55.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:43:55.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:55.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:56.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:57.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:57.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:57.499 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:43:57.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:43:57.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:57.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:57.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:58.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:58.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:58.950 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:43:58.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:43:58.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:58.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:59.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:00.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:00.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:00.620 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:44:00.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:44:00.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:00.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:01.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:02.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:02.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:02.783 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:44:02.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:44:02.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:02.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:03.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:04.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:04.460 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:44:04.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:04.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:44:04.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:04.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:04.821 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/40116,v1:127.0.0.1:6803/40116] [v2:127.0.0.1:6804/40116,v1:127.0.0.1:6805/40116] exists,up 9929830f-c5ed-44e5-8cfd-4f55651f0ef4 2024-09-15T02:44:04.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:44:04.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:44:04.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:44:04.822 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:44:04.822 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:44:04.822 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:44:04.823 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:44:04.823 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:04.823 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:44:04.823 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:44:04.823 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:44:04.823 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-15T02:44:04.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-09-15T02:44:04.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-09-15T02:44:04.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-09-15T02:44:04.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-15T02:44:05.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-15T02:44:07.518 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:44:07.518 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:44:07.518 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:44:07.519 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:44:07.519 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:07.519 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:44:07.519 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:44:07.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:44:07.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-15T02:44:07.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-09-15T02:44:07.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-09-15T02:44:07.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-15T02:44:08.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-15T02:44:08.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27209/ceph-osd.0.asok 2024-09-15T02:44:10.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-15T02:44:10.481 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.27209/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:44:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:44:10.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:44:10.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:44:10.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:44:10.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:44:10.676 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:44:10.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:44:10.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:44:10.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:44:10.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:44:10.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:44:10.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:44:10.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:44:10.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:44:10.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:44:10.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:10.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:44:10.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:44:10.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:44:10.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:44:10.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:44:10.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:44:10.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:44:10.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:44:10.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:44:10.690 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:44:10.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:44:10.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:44:10.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:44:10.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:44:10.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:44:10.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:44:10.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:44:10.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:44:10.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:44:10.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:10.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:44:10.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:44:10.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:44:10.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:44:10.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/mclock-config 0 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:44:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:44:10.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:44:10.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:44:10.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:44:10.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:44:10.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:44:10.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:44:10.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:44:10.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:44:10.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:44:10.703 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:44:10.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T02:44:10.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-15T02:44:10.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:44:10.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:10.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27209 2024-09-15T02:44:10.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27209 2024-09-15T02:44:10.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:44:10.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:44:10.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T02:44:10.708 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:44:10.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:44:10.774 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-09-15T02:44:10.774 DEBUG:teuthology.orchestra.run.smithi137:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-09-15T02:44:10.836 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:44:10.842 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T02:44:10.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/network-ping 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none ' 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-09-15T02:44:10.844 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-09-15T02:44:10.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T02:44:10.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-09-15T02:44:10.846 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_network_ping_test2' 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:44:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:44:10.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:44:10.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:44:10.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:44:10.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:44:10.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:44:10.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:44:10.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:44:10.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:44:10.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:44:10.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:44:10.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:44:10.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:44:10.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-15T02:44:10.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:44:10.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:10.853 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:10.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41201 2024-09-15T02:44:10.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:44:10.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:44:10.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-15T02:44:10.854 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:44:10.854 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:10.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:10.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.41201 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:44:10.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-15T02:44:10.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-15T02:44:11.209 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:44:11.209 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:44:11.209 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:44:11.210 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:44:11.210 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:11.210 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:11.210 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:44:11.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:44:11.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:44:11.452 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:44:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:44:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mon.a.asok 2024-09-15T02:44:11.453 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:44:11.453 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41201/ceph-mon.a.asok config get fsid 2024-09-15T02:44:11.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:44:11.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:44:11.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:44:11.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:44:11.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:44:11.529 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mon.a.asok 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:44:11.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41201/ceph-mon.a.asok config get mon_host 2024-09-15T02:44:11.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-09-15T02:44:11.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-15T02:44:11.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:44:11.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:44:11.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:44:11.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-15T02:44:11.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:44:12.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:44:12.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:44:12.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:44:12.437 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:44:12.437 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:12.437 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:12.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:44:12.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:44:12.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:44:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-09-15T02:44:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-15T02:44:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:44:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:44:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:44:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-15T02:44:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-15T02:44:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:44:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:44:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:44:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-15T02:44:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-15T02:44:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:44:12.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:44:12.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-15T02:44:12.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:44:12.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:44:12.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:44:12.469 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:44:12.469 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-15T02:44:12.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-15T02:44:12.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:44:12.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:44:12.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:44:12.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:44:12.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:44:12.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:44:12.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-15T02:44:12.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:44:12.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25c7fa43-3270-428f-80de-4601f32fccec 2024-09-15T02:44:12.472 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 25c7fa43-3270-428f-80de-4601f32fccec 2024-09-15T02:44:12.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 25c7fa43-3270-428f-80de-4601f32fccec' 2024-09-15T02:44:12.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:44:12.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8SeZmNNOUHBAAdkmxwFh3IauSePtMatvwuQ== 2024-09-15T02:44:12.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8SeZmNNOUHBAAdkmxwFh3IauSePtMatvwuQ=="}' 2024-09-15T02:44:12.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25c7fa43-3270-428f-80de-4601f32fccec -i td/network-ping/0/new.json 2024-09-15T02:44:12.781 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:44:12.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-15T02:44:12.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8SeZmNNOUHBAAdkmxwFh3IauSePtMatvwuQ== --osd-uuid 25c7fa43-3270-428f-80de-4601f32fccec 2024-09-15T02:44:12.816 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:12.814+0000 7fc7acc4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:12.826 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:12.825+0000 7fc7acc4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:12.827 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:12.826+0000 7fc7acc4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:12.827 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:12.826+0000 7fc7acc4f440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-15T02:44:12.828 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:12.826+0000 7fc7acc4f440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-15T02:44:12.828 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:12.826+0000 7fc7acc4f440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-15T02:44:12.867 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:12.865+0000 7fc7acc4f440 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-15T02:44:18.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-15T02:44:18.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:44:18.239 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:44:18.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:44:18.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:44:18.728 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:44:18.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:44:18.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:44:18.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:44:18.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:44:18.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:44:18.752 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:18.751+0000 7f28ed186440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:18.754 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:18.753+0000 7f28ed186440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:18.755 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:18.754+0000 7f28ed186440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:19.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:44:19.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:44:19.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:44:19.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:44:19.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:44:19.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:19.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:44:19.080 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:44:19.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:19.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:19.271 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:19.270+0000 7f28ed186440 -1 Falling back to public interface 2024-09-15T02:44:19.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:20.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:20.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:20.433 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:44:20.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:44:20.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:20.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:20.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:21.760 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:21.760+0000 7f28ed186440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:44:21.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:21.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:21.823 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:44:21.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:44:21.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:21.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:22.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:23.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:23.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:23.338 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:44:23.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:44:23.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:23.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:23.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:24.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:24.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:24.716 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:44:24.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:44:24.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:24.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:25.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:26.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:26.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:26.123 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:44:26.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:44:26.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:26.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:26.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:27.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:27.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:27.604 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:44:27.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:44:27.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:27.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:27.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:28.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:28.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:28.987 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:44:28.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:44:28.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:28.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:29.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:30.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:30.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:30.426 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:44:30.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:44:30.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:30.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:30.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:31.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:31.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:31.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:44:31.789 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:44:31.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:31.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:32.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:33.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:33.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:33.177 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:44:33.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:44:33.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:33.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:34.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:35.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:35.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:35.097 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:44:35.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:44:35.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:35.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:35.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:36.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:36.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:36.562 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:44:36.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:44:36.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:36.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:37.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:38.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:38.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:38.277 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:44:38.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:44:38.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:38.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:39.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:40.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:40.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:40.004 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:44:40.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:44:40.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:40.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:40.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:41.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:41.561 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:44:41.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:41.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:44:41.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:41.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:42.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:43.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:43.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:43.222 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:44:43.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:44:43.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:43.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:44:43.829 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42292,v1:127.0.0.1:6803/42292] [v2:127.0.0.1:6804/42292,v1:127.0.0.1:6805/42292] exists,up 25c7fa43-3270-428f-80de-4601f32fccec 2024-09-15T02:44:43.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:44:43.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:44:43.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:44:43.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:44:43.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:44:43.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:44:43.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd020e88-32ff-4059-8836-534052e5be00 2024-09-15T02:44:43.836 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 cd020e88-32ff-4059-8836-534052e5be00 2024-09-15T02:44:43.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cd020e88-32ff-4059-8836-534052e5be00' 2024-09-15T02:44:43.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:44:43.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbSuZmvbNwMhAAYU5jCERn9KodxkNzf6Iudg== 2024-09-15T02:44:43.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbSuZmvbNwMhAAYU5jCERn9KodxkNzf6Iudg=="}' 2024-09-15T02:44:43.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd020e88-32ff-4059-8836-534052e5be00 -i td/network-ping/1/new.json 2024-09-15T02:44:44.322 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:44:44.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-15T02:44:44.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbSuZmvbNwMhAAYU5jCERn9KodxkNzf6Iudg== --osd-uuid cd020e88-32ff-4059-8836-534052e5be00 2024-09-15T02:44:44.358 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:44.357+0000 7ff30200a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:44.360 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:44.359+0000 7ff30200a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:44.361 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:44.360+0000 7ff30200a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:44.361 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:44.360+0000 7ff30200a440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-15T02:44:44.362 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:44.360+0000 7ff30200a440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-15T02:44:44.362 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:44.360+0000 7ff30200a440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-15T02:44:44.404 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:44.402+0000 7ff30200a440 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-15T02:44:50.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-15T02:44:50.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:44:50.223 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T02:44:50.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:44:50.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:44:50.721 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T02:44:50.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:44:50.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:44:50.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:44:50.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:44:50.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:44:50.744 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:50.742+0000 7f541d60c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:50.770 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:50.769+0000 7f541d60c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:50.771 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:50.770+0000 7f541d60c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:44:51.100 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:44:51.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:51.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:51.286 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:51.284+0000 7f541d60c440 -1 Falling back to public interface 2024-09-15T02:44:51.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:52.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:52.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:52.493 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:44:52.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:44:52.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:52.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:52.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:53.773 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:44:53.772+0000 7f541d60c440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:44:53.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:53.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:53.928 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:44:53.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:44:53.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:53.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:54.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:55.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:55.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:55.427 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:44:55.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:44:55.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:55.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:55.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:56.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:56.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:56.856 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:44:56.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:44:56.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:56.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:57.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:58.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:58.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:58.322 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:44:58.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:44:58.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:58.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:58.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:59.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:59.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:59.795 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:44:59.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:44:59.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:59.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:00.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:01.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:01.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:01.215 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:45:01.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:45:01.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:01.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:01.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:02.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:02.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:02.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:45:02.707 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:45:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:03.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:04.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:04.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:04.119 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:45:04.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:45:04.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:04.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:04.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:05.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:05.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:05.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:45:05.625 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:45:05.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:05.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:06.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:07.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:07.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:07.138 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:45:07.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:45:07.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:07.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:07.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:08.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:08.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:08.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:45:08.906 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:45:08.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:08.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:09.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:10.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:10.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:10.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:45:10.717 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:45:10.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:10.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:12.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:45:12.467 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:45:12.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:12.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:13.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:14.213 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:45:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:45:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:14.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:15.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:15.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:15.647 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:45:15.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:45:15.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:15.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:16.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:17.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:17.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:17.151 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:45:17.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:45:17.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:17.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:45:17.505 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/44021,v1:127.0.0.1:6811/44021] [v2:127.0.0.1:6812/44021,v1:127.0.0.1:6813/44021] exists,up cd020e88-32ff-4059-8836-534052e5be00 2024-09-15T02:45:17.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:45:17.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:45:17.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-15T02:45:17.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:45:17.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-15T02:45:17.512 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:45:17.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38b363e7-0635-41cc-b8c9-f0b0ae142207 2024-09-15T02:45:17.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 38b363e7-0635-41cc-b8c9-f0b0ae142207' 2024-09-15T02:45:17.513 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 38b363e7-0635-41cc-b8c9-f0b0ae142207 2024-09-15T02:45:17.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:45:17.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9SuZm0iI2HxAArdrrcS/WoTgJDHX/vPZNUw== 2024-09-15T02:45:17.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9SuZm0iI2HxAArdrrcS/WoTgJDHX/vPZNUw=="}' 2024-09-15T02:45:17.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38b363e7-0635-41cc-b8c9-f0b0ae142207 -i td/network-ping/2/new.json 2024-09-15T02:45:18.131 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:45:18.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-15T02:45:18.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9SuZm0iI2HxAArdrrcS/WoTgJDHX/vPZNUw== --osd-uuid 38b363e7-0635-41cc-b8c9-f0b0ae142207 2024-09-15T02:45:18.168 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:18.166+0000 7fae36a2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:18.190 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:18.189+0000 7fae36a2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:18.191 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:18.190+0000 7fae36a2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:18.192 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:18.190+0000 7fae36a2d440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-15T02:45:18.192 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:18.190+0000 7fae36a2d440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-15T02:45:18.192 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:18.190+0000 7fae36a2d440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-15T02:45:18.282 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:18.281+0000 7fae36a2d440 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-15T02:45:23.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-15T02:45:23.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:45:23.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:45:23.974 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T02:45:23.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:45:24.460 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T02:45:24.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:45:24.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:45:24.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:45:24.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:45:24.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:45:24.484 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:24.483+0000 7f4e5057b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:24.485 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:24.485+0000 7f4e5057b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:24.487 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:24.486+0000 7f4e5057b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:45:24.809 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:45:24.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:24.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:25.000 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:25.000+0000 7f4e5057b440 -1 Falling back to public interface 2024-09-15T02:45:25.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:26.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:26.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:26.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:45:26.161 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:45:26.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:26.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:26.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:27.491 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:45:27.490+0000 7f4e5057b440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:45:27.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:27.519 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:45:27.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:27.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:45:27.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:27.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:27.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:28.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:28.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:28.993 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:45:28.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:45:28.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:28.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:29.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:30.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:30.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:30.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:45:30.465 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:45:30.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:30.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:30.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:31.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:31.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:31.888 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:45:31.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:45:31.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:31.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:32.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:33.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:33.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:33.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:45:33.252 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:45:33.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:33.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:33.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:34.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:34.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:34.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:45:34.610 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:45:34.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:34.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:34.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:35.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:35.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:35.972 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:45:35.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:45:35.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:35.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:36.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:37.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:37.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:37.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:45:37.446 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:45:37.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:37.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:37.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:38.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:38.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:38.833 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:45:38.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:45:38.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:38.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:39.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:40.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:40.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:40.359 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:45:40.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:45:40.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:40.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:42.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:42.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:42.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:45:42.310 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:45:42.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:42.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:43.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:44.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:44.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:44.110 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:45:44.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:45:44.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:44.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:44.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:45.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:45.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:45.520 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:45:45.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:45:45.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:45.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:46.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:47.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:47.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:47.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:45:47.192 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:45:47.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:47.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:48.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:49.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:49.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:49.103 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:45:49.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:45:49.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:49.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:50.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:50.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:50.941 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:45:50.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:45:50.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:50.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:45:51.294 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/45775,v1:127.0.0.1:6819/45775] [v2:127.0.0.1:6820/45775,v1:127.0.0.1:6821/45775] exists,up 38b363e7-0635-41cc-b8c9-f0b0ae142207 2024-09-15T02:45:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:45:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:45:51.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:45:51.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-09-15T02:45:56.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-09-15T02:45:56.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-15T02:45:56.792 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'foo' already exists 2024-09-15T02:45:56.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:45:57.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-15T02:45:57.849 INFO:tasks.workunit.client.0.smithi137.stdout:hints = 1 2024-09-15T02:45:57.849 INFO:tasks.workunit.client.0.smithi137.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-15T02:45:57.849 INFO:tasks.workunit.client.0.smithi137.stdout:Object prefix: benchmark_data_smithi137_46715 2024-09-15T02:45:57.849 INFO:tasks.workunit.client.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T02:45:57.849 INFO:tasks.workunit.client.0.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-15T02:45:58.849 INFO:tasks.workunit.client.0.smithi137.stdout: 1 16 64 48 0.187474 0.1875 0.255081 0.281102 2024-09-15T02:45:59.849 INFO:tasks.workunit.client.0.smithi137.stdout: 2 16 131 115 0.224575 0.261719 0.232732 0.264359 2024-09-15T02:46:00.850 INFO:tasks.workunit.client.0.smithi137.stdout: 3 16 195 179 0.233041 0.25 0.236307 0.257027 2024-09-15T02:46:01.849 INFO:tasks.workunit.client.0.smithi137.stdout: 4 16 259 243 0.237272 0.25 0.254564 0.252945 2024-09-15T02:46:02.850 INFO:tasks.workunit.client.0.smithi137.stdout: 5 16 330 314 0.24528 0.277344 0.251067 0.250285 2024-09-15T02:46:03.850 INFO:tasks.workunit.client.0.smithi137.stdout: 6 16 394 378 0.246061 0.25 0.222101 0.247191 2024-09-15T02:46:04.851 INFO:tasks.workunit.client.0.smithi137.stdout: 7 16 467 451 0.251642 0.285156 0.251909 0.245616 2024-09-15T02:46:05.850 INFO:tasks.workunit.client.0.smithi137.stdout: 8 16 531 515 0.251432 0.25 0.229981 0.244295 2024-09-15T02:46:06.850 INFO:tasks.workunit.client.0.smithi137.stdout: 9 16 598 582 0.252572 0.261719 0.293454 0.243736 2024-09-15T02:46:07.850 INFO:tasks.workunit.client.0.smithi137.stdout: 10 16 675 659 0.25739 0.300781 0.225911 0.241485 2024-09-15T02:46:08.850 INFO:tasks.workunit.client.0.smithi137.stdout:Total time run: 10.2144 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Total writes made: 675 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Write size: 4096 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Object size: 4096 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Bandwidth (MB/sec): 0.258137 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Stddev Bandwidth: 0.0301707 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Max bandwidth (MB/sec): 0.300781 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Min bandwidth (MB/sec): 0.1875 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Average IOPS: 66 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Stddev IOPS: 7.7237 2024-09-15T02:46:08.851 INFO:tasks.workunit.client.0.smithi137.stdout:Max IOPS: 77 2024-09-15T02:46:08.852 INFO:tasks.workunit.client.0.smithi137.stdout:Min IOPS: 48 2024-09-15T02:46:08.852 INFO:tasks.workunit.client.0.smithi137.stdout:Average Latency(s): 0.24086 2024-09-15T02:46:08.852 INFO:tasks.workunit.client.0.smithi137.stdout:Stddev Latency(s): 0.0304671 2024-09-15T02:46:08.852 INFO:tasks.workunit.client.0.smithi137.stdout:Max latency(s): 0.408666 2024-09-15T02:46:08.852 INFO:tasks.workunit.client.0.smithi137.stdout:Min latency(s): 0.205472 2024-09-15T02:46:08.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-09-15T02:46:18.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-09-15T02:46:18.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:46:18.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:46:19.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:46:19.327 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:46:19.327 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T02:46:19.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:46:19.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:46:19.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:46:19.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-15T02:46:19.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-15T02:46:19.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-15T02:46:19.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:46:19.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:46:19.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-15T02:46:19.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-15T02:46:19.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672975' 2024-09-15T02:46:19.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:46:19.562 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:46:19.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-15T02:46:19.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-15T02:46:19.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672975 2-64424509451' 2024-09-15T02:46:19.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:46:19.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-15T02:46:19.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:46:19.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:46:19.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-15T02:46:19.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:46:19.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-15T02:46:19.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-15T02:46:19.669 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836507 2024-09-15T02:46:19.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:46:20.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836507 2024-09-15T02:46:20.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:46:21.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:46:21.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:46:21.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-15T02:46:21.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:46:21.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-15T02:46:21.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:46:21.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:46:21.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-15T02:46:21.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:46:21.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-15T02:46:21.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-15T02:46:21.372 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672975 2024-09-15T02:46:21.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:46:21.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-15T02:46:21.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:46:21.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-15T02:46:21.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:46:21.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:46:21.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-15T02:46:21.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:46:21.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-15T02:46:21.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T02:46:21.728 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509451 2024-09-15T02:46:21.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:46:22.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T02:46:22.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:22.078 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-09-15T02:46:22.078 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:46:22.079 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:46:22.079 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:22.079 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:22.079 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:22.079 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-osd.0.asok 2024-09-15T02:46:22.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:22.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-osd.0.asok dump_osd_network 2024-09-15T02:46:22.149 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:22.149 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 1000, 2024-09-15T02:46:22.149 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:46:22.149 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:22.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:22.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:22.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:22.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-09-15T02:46:22.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mgr.x.asok 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:22.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-mgr.x.asok dump_osd_network 2024-09-15T02:46:22.227 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:22.227 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 1000, 2024-09-15T02:46:22.227 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:46:22.227 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:22.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:22.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:22.237 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:22.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-09-15T02:46:22.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:22.241 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-09-15T02:46:22.241 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:46:22.241 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:46:22.242 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:22.242 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:22.242 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:22.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-osd.0.asok 2024-09-15T02:46:22.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:22.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-osd.0.asok dump_osd_network 0 2024-09-15T02:46:22.313 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 0, 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [ 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:21 2024", 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.303, 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.321, 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.297 2024-09-15T02:46:22.314 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.377, 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.448, 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.448 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.377, 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.448, 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.448 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.273 2024-09-15T02:46:22.315 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:21 2024", 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.297, 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.319, 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.299 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.316 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.411, 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.494, 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.494 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.411, 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.494, 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.494 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.244 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:21 2024", 2024-09-15T02:46:22.317 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.290, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.313, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.283 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.358, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.478, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.318 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.358, 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.478, 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.264 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:21 2024", 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.319 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.267, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.284, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.275 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.391, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.434, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.434 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.391, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.434, 2024-09-15T02:46:22.320 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.434 2024-09-15T02:46:22.321 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.321 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.229 2024-09-15T02:46:22.321 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T02:46:22.321 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T02:46:22.321 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:22.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:22.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-09-15T02:46:22.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:22.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:22.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:22.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-15T02:46:22.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-15T02:46:22.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-15T02:46:22.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:22.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:22.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:22.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mgr.x.asok 2024-09-15T02:46:22.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:22.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-mgr.x.asok dump_osd_network 0 2024-09-15T02:46:22.392 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:22.392 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 0, 2024-09-15T02:46:22.392 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [ 2024-09-15T02:46:22.392 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:16 2024", 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.364, 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.318, 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.296 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.297, 2024-09-15T02:46:22.393 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.185, 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.185 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.448, 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.448, 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.448 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.284 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:16 2024", 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.394 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.360, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.313, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.300 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.256, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.156, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.156 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.395 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.494, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.494, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.494 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.247 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:16 2024", 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.359, 2024-09-15T02:46:22.396 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.309, 2024-09-15T02:46:22.397 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.282 2024-09-15T02:46:22.397 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.397 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.397 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.287, 2024-09-15T02:46:22.397 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.171, 2024-09-15T02:46:22.397 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.171 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.478, 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.478, 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.268 2024-09-15T02:46:22.398 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:20 2024", 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.312, 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.321, 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.333 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.399 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.203, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.202, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.202 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.460, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.500, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.510 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.265 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:16 2024", 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.400 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.333, 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.278, 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.277 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.239, 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.135, 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.135 2024-09-15T02:46:22.401 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.434, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.434, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.434 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.218 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:20 2024", 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.402 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.307, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.310, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.330 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.226, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.194, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.194 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.443, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.462, 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.497 2024-09-15T02:46:22.403 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.251 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:15 2024", 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.330, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.308, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.312 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.404 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.259, 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.206, 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.206 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.462, 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.462, 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.462 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.368 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.405 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:20 2024", 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.293, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.293, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.319 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.212, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.184, 2024-09-15T02:46:22.406 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.184 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.424, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.477, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.479 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.235 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:20 2024", 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.407 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.279, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.295, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.318 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.179, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.179, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.179 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.385, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.462, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.462 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.408 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.206 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:16 2024", 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.311, 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.284, 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.256 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.409 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.232, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.184, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.162 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.424, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.466, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.466 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.331 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:15 2024", 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:22.410 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.310, 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.287, 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.296 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.249, 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.200, 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.200 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.411 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.411, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.411, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.411 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.305 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:16 2024", 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.278, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.257, 2024-09-15T02:46:22.412 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.237 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.167, 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.155, 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.136 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.443, 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.443, 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.443 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.345 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T02:46:22.413 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T02:46:22.414 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:22.414 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.415 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:22.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-09-15T02:46:22.415 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:22.415 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:22.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:22.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-09-15T02:46:42.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-09-15T02:46:42.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:46:42.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:46:42.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:46:42.802 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:46:42.802 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T02:46:42.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:46:42.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:46:42.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:46:42.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836513 2024-09-15T02:46:42.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836513 2024-09-15T02:46:42.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513' 2024-09-15T02:46:42.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:46:42.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:46:43.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-15T02:46:43.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-15T02:46:43.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672982' 2024-09-15T02:46:43.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:46:43.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:46:43.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509458 2024-09-15T02:46:43.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509458 2024-09-15T02:46:43.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672982 2-64424509458' 2024-09-15T02:46:43.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:46:43.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836513 2024-09-15T02:46:43.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:46:43.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:46:43.119 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836513 2024-09-15T02:46:43.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:46:43.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836513 2024-09-15T02:46:43.121 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836513 2024-09-15T02:46:43.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-15T02:46:43.121 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:46:43.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836513 2024-09-15T02:46:43.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:46:44.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:46:44.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:46:44.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-15T02:46:44.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:46:44.835 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-15T02:46:44.835 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:46:44.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:46:44.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-15T02:46:44.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:46:44.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-15T02:46:44.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-15T02:46:44.837 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672982 2024-09-15T02:46:44.838 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:46:45.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-15T02:46:45.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:46:45.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509458 2024-09-15T02:46:45.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:46:45.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:46:45.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509458 2024-09-15T02:46:45.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:46:45.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509458 2024-09-15T02:46:45.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-15T02:46:45.201 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509458 2024-09-15T02:46:45.201 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:46:45.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-15T02:46:45.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:45.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-osd.0.asok 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:45.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-osd.0.asok dump_osd_network 2024-09-15T02:46:45.626 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:45.626 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 1000, 2024-09-15T02:46:45.626 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:46:45.626 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:45.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:45.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:45.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.637 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:45.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-09-15T02:46:45.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:45.640 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-15T02:46:45.640 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-15T02:46:45.640 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-15T02:46:45.640 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:45.641 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:45.641 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:45.641 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mgr.x.asok 2024-09-15T02:46:45.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:45.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-mgr.x.asok dump_osd_network 2024-09-15T02:46:45.705 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:45.705 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 1000, 2024-09-15T02:46:45.705 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:46:45.705 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:45.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:45.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:45.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:45.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-09-15T02:46:45.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:45.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-09-15T02:46:45.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:46:45.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:46:45.722 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:45.722 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:45.722 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:45.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-osd.0.asok 2024-09-15T02:46:45.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:45.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-osd.0.asok dump_osd_network 0 2024-09-15T02:46:45.792 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:45.793 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 0, 2024-09-15T02:46:45.793 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [ 2024-09-15T02:46:45.793 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.793 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.793 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.323, 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.330, 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.311 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.794 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.406, 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.439, 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.448 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.406, 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.439, 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.448 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.795 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.194 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.796 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.288, 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.313, 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.297 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.364, 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.428, 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:46:45.797 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.364, 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.428, 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.238 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.798 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.299, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.305, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.299 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.377, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.411, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.494 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.799 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.377, 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.411, 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.494 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.226 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.800 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.268, 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.280, 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.273 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.425, 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.425, 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.434 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.801 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.425, 2024-09-15T02:46:45.802 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.425, 2024-09-15T02:46:45.802 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.434 2024-09-15T02:46:45.802 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.802 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.215 2024-09-15T02:46:45.802 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T02:46:45.802 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T02:46:45.802 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:45.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:45.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-09-15T02:46:45.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:45.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:45.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:46:45.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-15T02:46:45.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-15T02:46:45.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-15T02:46:45.809 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:46:45.809 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:45.809 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:46:45.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mgr.x.asok 2024-09-15T02:46:45.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:46:45.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-mgr.x.asok dump_osd_network 0 2024-09-15T02:46:45.872 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:46:45.872 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 0, 2024-09-15T02:46:45.872 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [ 2024-09-15T02:46:45.872 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.872 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:40 2024", 2024-09-15T02:46:45.872 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.359, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.316, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.313 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.248, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.215, 2024-09-15T02:46:45.873 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.206 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.463, 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.463, 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.463 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.217 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:40 2024", 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.874 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.335, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.292, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.293 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.227, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.171, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.171 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.875 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.876 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.456, 2024-09-15T02:46:45.876 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.456, 2024-09-15T02:46:45.876 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.456 2024-09-15T02:46:45.876 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.876 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.187 2024-09-15T02:46:45.876 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.877 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.877 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.877 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.877 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.877 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:45.877 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.323, 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.330, 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.311 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.260, 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.255, 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.185 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.406, 2024-09-15T02:46:45.878 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.439, 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.448 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.259 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:40 2024", 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.299, 2024-09-15T02:46:45.879 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.302, 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.322 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.274, 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.203, 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.202 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.350, 2024-09-15T02:46:45.880 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.460, 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.510 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.310 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:40 2024", 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.881 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.285, 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.302, 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.320 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.259, 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.200, 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.194 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.882 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.324, 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.443, 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.497 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.251 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.883 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.288, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.313, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.297 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.248, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.248, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.171 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.884 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.364, 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.428, 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.221 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:40 2024", 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:45.885 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.262, 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.288, 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.307 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.229, 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.212, 2024-09-15T02:46:45.886 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.184 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.281, 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.424, 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.479 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.262 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.887 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:40 2024", 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.268, 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.287, 2024-09-15T02:46:45.888 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.307 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.210, 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.179, 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.179 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.343, 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.385, 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.462 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.273 2024-09-15T02:46:45.889 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.299, 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.305, 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.299 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.890 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.241, 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.224, 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.156 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.377, 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.411, 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.494 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.231 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.891 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.302, 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.293, 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.283 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.892 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.213, 2024-09-15T02:46:45.893 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.187, 2024-09-15T02:46:45.893 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.173 2024-09-15T02:46:45.893 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.893 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.893 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.428, 2024-09-15T02:46:45.893 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.478, 2024-09-15T02:46:45.893 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:46:45.894 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.894 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.198 2024-09-15T02:46:45.894 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.894 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.894 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.894 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.894 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.298, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.286, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.264 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.197, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.149, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.136 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.895 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.416, 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.461, 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.461 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.229 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:46:41 2024", 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:46:45.896 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.268, 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.280, 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.273 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.208, 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.208, 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.135 2024-09-15T02:46:45.897 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.425, 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.425, 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.434 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.185 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T02:46:45.898 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:46:45.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:46:45.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-09-15T02:46:45.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:46:45.900 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-09-15T02:46:45.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:46:45.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-09-15T02:47:35.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-09-15T02:47:35.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:47:35.888 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:47:36.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:47:36.247 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:47:36.248 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T02:47:36.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:47:36.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:36.248 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:47:36.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836526 2024-09-15T02:47:36.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836526 2024-09-15T02:47:36.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526' 2024-09-15T02:47:36.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:36.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:47:36.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672994 2024-09-15T02:47:36.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672994 2024-09-15T02:47:36.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949672994' 2024-09-15T02:47:36.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:36.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:47:36.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509471 2024-09-15T02:47:36.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509471 2024-09-15T02:47:36.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949672994 2-64424509471' 2024-09-15T02:47:36.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:36.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836526 2024-09-15T02:47:36.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:36.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:47:36.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836526 2024-09-15T02:47:36.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:36.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836526 2024-09-15T02:47:36.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-15T02:47:36.631 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836526 2024-09-15T02:47:36.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:47:36.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836526 2024-09-15T02:47:36.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:47:37.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:47:37.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:47:38.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-15T02:47:38.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:38.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672994 2024-09-15T02:47:38.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:38.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:47:38.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672994 2024-09-15T02:47:38.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:38.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672994 2024-09-15T02:47:38.358 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672994 2024-09-15T02:47:38.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-09-15T02:47:38.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:47:38.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672995 -lt 42949672994 2024-09-15T02:47:38.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:38.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509471 2024-09-15T02:47:38.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:38.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:47:38.733 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509471 2024-09-15T02:47:38.733 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:38.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509471 2024-09-15T02:47:38.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-15T02:47:38.734 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509471 2024-09-15T02:47:38.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:47:39.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509471 -lt 64424509471 2024-09-15T02:47:39.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:47:39.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-09-15T02:47:39.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:47:39.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:47:39.101 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:39.101 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.101 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-osd.0.asok 2024-09-15T02:47:39.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:47:39.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-osd.0.asok dump_osd_network 2024-09-15T02:47:39.174 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:47:39.174 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 1000, 2024-09-15T02:47:39.174 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:47:39.174 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:47:39.181 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.182 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:47:39.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:47:39.185 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.185 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-09-15T02:47:39.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-09-15T02:47:39.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:47:39.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-15T02:47:39.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-15T02:47:39.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-15T02:47:39.188 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:39.188 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.189 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.189 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mgr.x.asok 2024-09-15T02:47:39.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:47:39.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-mgr.x.asok dump_osd_network 2024-09-15T02:47:39.253 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:47:39.253 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 1000, 2024-09-15T02:47:39.253 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:47:39.253 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:47:39.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:47:39.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:47:39.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-09-15T02:47:39.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-09-15T02:47:39.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:47:39.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-09-15T02:47:39.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:47:39.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:47:39.267 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:39.267 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.267 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-osd.0.asok 2024-09-15T02:47:39.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:47:39.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-osd.0.asok dump_osd_network 0 2024-09-15T02:47:39.340 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:47:39.340 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 0, 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [ 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:36 2024", 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.345, 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.341, 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.331 2024-09-15T02:47:39.341 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.433, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.456, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.470 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.433, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.456, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.470 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.213 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.342 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:36 2024", 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.326, 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.315, 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.312 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.413, 2024-09-15T02:47:39.343 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.432, 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.459 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.413, 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.432, 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.459 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.197 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:36 2024", 2024-09-15T02:47:39.344 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.308, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.312, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.306 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.397, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.423, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.474 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.345 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.397, 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.423, 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.474 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.184 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:36 2024", 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.346 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.306, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.303, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.290 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.448, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.448, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.451 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.448, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.448, 2024-09-15T02:47:39.347 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.451 2024-09-15T02:47:39.348 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.348 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.226 2024-09-15T02:47:39.348 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T02:47:39.348 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T02:47:39.348 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:47:39.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:47:39.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-09-15T02:47:39.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-09-15T02:47:39.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:47:39.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:47:39.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-15T02:47:39.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-15T02:47:39.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-15T02:47:39.356 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:39.356 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.356 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mgr.x.asok 2024-09-15T02:47:39.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:47:39.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-mgr.x.asok dump_osd_network 0 2024-09-15T02:47:39.419 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:47:39.419 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 0, 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [ 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:35 2024", 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.374, 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.320, 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.308 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.420 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.277, 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.227, 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.163 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.487, 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.487, 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.487 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.227 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.421 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:35 2024", 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.351, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.336, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.318 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.271, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.232, 2024-09-15T02:47:39.422 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.216 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.495, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.495, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.495 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.273 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:35 2024", 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:47:39.423 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.349, 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.298, 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.294 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.268, 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.196, 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.171 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.424 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.475, 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.475, 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.475 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.212 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:31 2024", 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:47:39.425 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.341, 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.340, 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.328 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.285, 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.250, 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.194 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.425, 2024-09-15T02:47:39.426 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.456, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.470 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.417 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:35 2024", 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.328, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.321, 2024-09-15T02:47:39.427 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.309 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.256, 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.229, 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.188 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.476, 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.476, 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.476 2024-09-15T02:47:39.428 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.229 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:35 2024", 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.326, 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.317, 2024-09-15T02:47:39.429 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.300 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.240, 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.225, 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.174 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.430 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.455, 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.455, 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.455 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.215 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:35 2024", 2024-09-15T02:47:39.431 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.432 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 2, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.324, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.325, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.300 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.255, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.241, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.180 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.433 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.432, 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.432, 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.432 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.245 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:31 2024", 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.434 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 2, 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.307, 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.317, 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.309 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.222, 2024-09-15T02:47:39.435 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.222, 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.199 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.407, 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.459, 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.459 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.407 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:31 2024", 2024-09-15T02:47:39.436 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.314, 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.313, 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.306 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.437 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.252, 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.231, 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.184 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.383, 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.474, 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.474 2024-09-15T02:47:39.438 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.388 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:31 2024", 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 0, 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 1, 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.294, 2024-09-15T02:47:39.439 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.302, 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.287 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.225, 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.221, 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.167 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.350, 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.436, 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.451 2024-09-15T02:47:39.440 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.375 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:36 2024", 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "front", 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.292, 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.298, 2024-09-15T02:47:39.441 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.286 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.190, 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.190, 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.140 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.455, 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.455, 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.478 2024-09-15T02:47:39.442 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.190 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "last update": "Sun Sep 15 02:47:36 2024", 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "stale": false, 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "from osd": 1, 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "to osd": 0, 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "interface": "back", 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "average": { 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.288, 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.272, 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.277 2024-09-15T02:47:39.443 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "min": { 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.167, 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.167, 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.166 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "max": { 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "1min": 0.420, 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "5min": 0.420, 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "15min": 0.461 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: "last": 0.167 2024-09-15T02:47:39.444 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T02:47:39.445 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T02:47:39.445 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:47:39.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mgr.x.asok 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:47:39.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-mgr.x.asok dump_osd_network 99 2024-09-15T02:47:39.498 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:47:39.498 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 99, 2024-09-15T02:47:39.498 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:47:39.498 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:47:39.505 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.505 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-09-15T02:47:39.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-09-15T02:47:39.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-osd.0.asok 2024-09-15T02:47:39.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-09-15T02:47:39.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41201/ceph-osd.0.asok dump_osd_network 98 2024-09-15T02:47:39.581 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T02:47:39.581 INFO:tasks.workunit.client.0.smithi137.stdout: "threshold": 98, 2024-09-15T02:47:39.581 INFO:tasks.workunit.client.0.smithi137.stdout: "entries": [] 2024-09-15T02:47:39.581 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T02:47:39.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-09-15T02:47:39.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-09-15T02:47:39.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-09-15T02:47:39.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-09-15T02:47:39.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-15T02:47:39.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-15T02:47:39.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:47:39.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-15T02:47:39.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:47:39.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:47:39.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:47:39.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:47:39.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:47:39.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:47:39.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:47:39.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:47:39.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:47:39.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:47:39.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:47:39.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:47:39.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:47:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:47:39.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:47:39.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:47:39.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-15T02:47:39.724 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:47:39.724 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.724 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41201 2024-09-15T02:47:39.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:47:39.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:47:39.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-15T02:47:39.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-15T02:47:39.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:47:39.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:47:39.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:47:39.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:47:39.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:47:39.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:47:39.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:47:39.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:47:39.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:47:39.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:39.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:47:39.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:47:39.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:39.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:47:39.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:47:39.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-15T02:47:39.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:47:39.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41201 2024-09-15T02:47:39.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:47:39.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:47:39.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-15T02:47:39.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:47:39.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:39.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:39.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.41201 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:47:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:47:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:47:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-15T02:47:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-15T02:47:40.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:47:40.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:47:40.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:47:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:47:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:40.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:47:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:47:40.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:47:40.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:47:40.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:47:40.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:47:40.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:47:40.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:47:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:47:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:47:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:47:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mon.a.asok 2024-09-15T02:47:40.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:47:40.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41201/ceph-mon.a.asok config get fsid 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:47:40.470 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:47:40.471 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:40.471 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:40.471 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:40.471 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41201/ceph-mon.a.asok 2024-09-15T02:47:40.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:47:40.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41201/ceph-mon.a.asok config get mon_host 2024-09-15T02:47:40.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-09-15T02:47:40.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-15T02:47:40.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:47:40.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:47:40.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:47:40.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-15T02:47:40.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:47:41.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:47:41.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:47:41.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-09-15T02:47:41.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-15T02:47:41.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:47:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:47:41.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-15T02:47:41.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:47:41.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=203fb23b-1495-4104-9699-8084af4f5134 2024-09-15T02:47:41.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 203fb23b-1495-4104-9699-8084af4f5134' 2024-09-15T02:47:41.409 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 203fb23b-1495-4104-9699-8084af4f5134 2024-09-15T02:47:41.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:47:41.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNSuZmicUKGRAA+MV77vLz+YsM1TOnTBgpDg== 2024-09-15T02:47:41.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNSuZmicUKGRAA+MV77vLz+YsM1TOnTBgpDg=="}' 2024-09-15T02:47:41.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 203fb23b-1495-4104-9699-8084af4f5134 -i td/network-ping/0/new.json 2024-09-15T02:47:41.721 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:47:41.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-15T02:47:41.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNSuZmicUKGRAA+MV77vLz+YsM1TOnTBgpDg== --osd-uuid 203fb23b-1495-4104-9699-8084af4f5134 2024-09-15T02:47:41.757 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:41.756+0000 7f305584d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:41.768 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:41.766+0000 7f305584d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:41.769 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:41.767+0000 7f305584d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:41.769 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:41.768+0000 7f305584d440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-15T02:47:41.769 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:41.768+0000 7f305584d440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-15T02:47:41.769 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:41.768+0000 7f305584d440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-15T02:47:41.808 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:41.807+0000 7f305584d440 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-15T02:47:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-15T02:47:47.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:47:47.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:47:47.128 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:47:47.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:47:47.611 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:47:47.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:47:47.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:47:47.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:47:47.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:47:47.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:47:47.635 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:47.633+0000 7febd4388440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:47.637 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:47.635+0000 7febd4388440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:47.638 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:47.637+0000 7febd4388440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:47.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:47:47.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:47:47.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:47:47.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:47:47.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:47:47.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:47.963 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:47:47.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:47:47.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:47.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:48.155 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:48.154+0000 7febd4388440 -1 Falling back to public interface 2024-09-15T02:47:48.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:49.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:49.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:49.313 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:47:49.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:47:49.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:49.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:49.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:50.635 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:47:50.633+0000 7febd4388440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:47:50.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:50.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:50.666 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:47:50.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:47:50.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:50.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:51.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:52.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:52.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:52.229 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:47:52.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:47:52.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:52.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:52.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:53.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:53.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:53.633 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:47:53.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:47:53.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:53.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:54.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:55.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:55.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:55.030 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:47:55.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:47:55.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:55.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:55.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:56.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:56.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:56.519 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:47:56.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:47:56.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:56.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:56.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:57.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:57.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:57.892 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:47:57.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:47:57.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:57.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:58.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:59.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:59.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:59.397 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:47:59.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:47:59.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:59.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:59.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:00.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:00.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:00.754 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:48:00.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:48:00.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:00.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:01.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:02.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:02.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:02.268 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:48:02.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:48:02.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:02.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:03.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:04.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:04.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:04.131 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:48:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:48:04.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:04.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:04.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:05.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:05.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:05.589 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:48:05.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:48:05.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:05.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:06.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:07.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:07.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:07.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:48:07.343 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:48:07.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:07.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:08.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:08.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:08.963 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:48:08.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:48:08.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:08.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:10.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:11.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:11.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:11.076 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:48:11.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:48:11.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:11.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:11.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:12.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:12.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:12.577 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:48:12.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:48:12.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:12.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:12.979 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/48603,v1:127.0.0.1:6803/48603] [v2:127.0.0.1:6804/48603,v1:127.0.0.1:6805/48603] exists,up 203fb23b-1495-4104-9699-8084af4f5134 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:48:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:48:12.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:48:12.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:48:12.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fee878cd-caee-45ec-8f4d-50cd23dc60df 2024-09-15T02:48:12.987 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 fee878cd-caee-45ec-8f4d-50cd23dc60df 2024-09-15T02:48:12.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fee878cd-caee-45ec-8f4d-50cd23dc60df' 2024-09-15T02:48:12.987 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:48:12.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsSuZmFc9pOxAAg+rZdLFVjv+s0lTLonBaWQ== 2024-09-15T02:48:12.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsSuZmFc9pOxAAg+rZdLFVjv+s0lTLonBaWQ=="}' 2024-09-15T02:48:12.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fee878cd-caee-45ec-8f4d-50cd23dc60df -i td/network-ping/1/new.json 2024-09-15T02:48:13.463 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:48:13.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-15T02:48:13.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsSuZmFc9pOxAAg+rZdLFVjv+s0lTLonBaWQ== --osd-uuid fee878cd-caee-45ec-8f4d-50cd23dc60df 2024-09-15T02:48:13.500 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:13.500+0000 7fc749d33440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:13.502 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:13.502+0000 7fc749d33440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:13.503 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:13.503+0000 7fc749d33440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:13.503 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:13.503+0000 7fc749d33440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-15T02:48:13.503 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:13.503+0000 7fc749d33440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-15T02:48:13.504 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:13.503+0000 7fc749d33440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-15T02:48:13.579 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:13.578+0000 7fc749d33440 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-15T02:48:19.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-15T02:48:19.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:48:19.405 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T02:48:19.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:48:19.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:48:19.888 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T02:48:19.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:48:19.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:48:19.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:48:19.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:48:19.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:48:19.911 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:19.911+0000 7f6afdc1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:19.913 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:19.913+0000 7f6afdc1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:19.914 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:19.914+0000 7f6afdc1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:20.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:48:20.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:48:20.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:48:20.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:48:20.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:48:20.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:20.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:48:20.237 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:48:20.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:20.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:20.439 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:20.439+0000 7f6afdc1e440 -1 Falling back to public interface 2024-09-15T02:48:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:21.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:21.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:21.589 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:48:21.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:48:21.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:21.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:21.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:22.931 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:22.930+0000 7f6afdc1e440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:48:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:48:22.940 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:48:22.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:22.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:23.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:24.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:24.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:24.401 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:48:24.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:48:24.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:24.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:24.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:25.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:25.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:25.834 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:48:25.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:48:25.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:25.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:26.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:27.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:27.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:27.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:48:27.194 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:48:27.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:27.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:27.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:28.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:28.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:28.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:48:28.700 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:48:28.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:28.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:29.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:30.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:30.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:30.122 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:48:30.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:48:30.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:30.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:30.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:31.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:31.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:31.519 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:48:31.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:48:31.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:31.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:32.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:33.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:33.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:33.027 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:48:33.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:48:33.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:33.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:33.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:34.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:34.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:34.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:48:34.412 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:48:34.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:34.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:35.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:36.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:36.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:36.404 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:48:36.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:48:36.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:36.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:37.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:38.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:38.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:38.207 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:48:38.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:48:38.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:38.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:38.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:39.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:39.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:39.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:48:39.987 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:48:39.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:39.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:40.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:41.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:41.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:41.586 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:48:41.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:48:41.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:41.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:42.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:43.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:43.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:43.018 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:48:43.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:48:43.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:43.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:43.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:44.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:44.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:44.427 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:48:44.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:48:44.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:44.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:45.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:46.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:46.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:46.104 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:48:46.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:48:46.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:46.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:46.463 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/50334,v1:127.0.0.1:6811/50334] [v2:127.0.0.1:6812/50334,v1:127.0.0.1:6813/50334] exists,up fee878cd-caee-45ec-8f4d-50cd23dc60df 2024-09-15T02:48:46.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:48:46.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:48:46.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:48:46.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-09-15T02:48:46.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-15T02:48:46.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' 2024-09-15T02:48:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-15T02:48:46.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:48:46.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4e825ebe-196d-4d85-b5a5-f240cb7df5d2 2024-09-15T02:48:46.471 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 4e825ebe-196d-4d85-b5a5-f240cb7df5d2 2024-09-15T02:48:46.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4e825ebe-196d-4d85-b5a5-f240cb7df5d2' 2024-09-15T02:48:46.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:48:46.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOS+ZmbrK5HBAAW6w+bTm5wx4+ZUpFY4XyXw== 2024-09-15T02:48:46.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOS+ZmbrK5HBAAW6w+bTm5wx4+ZUpFY4XyXw=="}' 2024-09-15T02:48:46.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4e825ebe-196d-4d85-b5a5-f240cb7df5d2 -i td/network-ping/2/new.json 2024-09-15T02:48:46.959 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:48:46.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-15T02:48:46.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOS+ZmbrK5HBAAW6w+bTm5wx4+ZUpFY4XyXw== --osd-uuid 4e825ebe-196d-4d85-b5a5-f240cb7df5d2 2024-09-15T02:48:46.997 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:46.995+0000 7f8458a34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:47.059 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:47.057+0000 7f8458a34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:47.060 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:47.058+0000 7f8458a34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:47.060 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:47.059+0000 7f8458a34440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-15T02:48:47.060 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:47.059+0000 7f8458a34440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-15T02:48:47.060 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:47.059+0000 7f8458a34440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-15T02:48:47.158 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:47.157+0000 7f8458a34440 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-15T02:48:53.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-15T02:48:53.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:48:53.009 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T02:48:53.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:48:53.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:48:53.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:48:53.485 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T02:48:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0e699283-a253-4134-9617-1f1c38351ade --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41201/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:48:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:48:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:48:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:48:53.509 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:53.507+0000 7f7f0e20f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:53.510 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:53.509+0000 7f7f0e20f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:53.511 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:53.510+0000 7f7f0e20f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:53.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:48:53.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:48:53.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:48:53.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:48:53.835 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:48:53.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:48:53.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:53.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:48:53.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:53.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:48:54.029 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:54.027+0000 7f7f0e20f440 -1 Falling back to public interface 2024-09-15T02:48:54.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:55.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:55.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:55.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:48:55.185 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:48:55.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:55.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:48:55.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:56.506 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:48:56.504+0000 7f7f0e20f440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:48:56.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:56.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:56.560 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:48:56.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:48:56.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:56.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:48:57.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:58.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:58.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:58.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:48:58.263 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:48:58.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:58.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:48:58.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:59.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:59.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:59.630 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:48:59.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:48:59.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:59.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:00.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:01.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:01.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:01.011 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:49:01.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:49:01.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:01.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:01.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:02.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:02.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:02.469 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:49:02.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:49:02.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:02.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:02.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:03.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:03.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:03.840 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:49:03.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:49:03.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:03.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:04.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:05.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:05.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:05.281 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:49:05.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:49:05.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:05.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:05.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:06.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:06.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:06.772 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:49:06.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:49:06.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:06.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:07.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:08.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:08.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:08.270 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:49:08.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:49:08.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:08.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:09.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:10.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:10.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:10.085 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:49:10.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:49:10.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:10.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:10.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:11.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:11.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:11.889 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:49:11.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:49:11.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:11.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:12.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:13.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:13.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:13.688 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:49:13.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:49:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:14.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:15.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:15.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:15.457 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:49:15.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:49:15.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:15.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:16.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:17.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:17.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:17.448 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:49:17.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:49:17.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:17.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:18.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:19.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:19.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:19.187 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:49:19.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:49:19.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:19.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:19.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:20.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:20.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:20.556 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:49:20.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:49:20.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:20.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:20.915 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/52088,v1:127.0.0.1:6819/52088] [v2:127.0.0.1:6820/52088,v1:127.0.0.1:6821/52088] exists,up 4e825ebe-196d-4d85-b5a5-f240cb7df5d2 2024-09-15T02:49:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:49:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:49:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:49:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-09-15T02:49:25.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-09-15T02:49:26.392 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'dc1' already exists 2024-09-15T02:49:26.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-09-15T02:49:27.014 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'dc2' already exists 2024-09-15T02:49:27.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-09-15T02:49:27.627 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'dc3' already exists 2024-09-15T02:49:27.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-09-15T02:49:28.160 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'rack1' already exists 2024-09-15T02:49:28.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-09-15T02:49:28.747 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'rack2' already exists 2024-09-15T02:49:28.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-09-15T02:49:29.349 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'rack3' already exists 2024-09-15T02:49:29.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-09-15T02:49:29.944 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'host1' already exists 2024-09-15T02:49:29.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-09-15T02:49:30.515 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'host2' already exists 2024-09-15T02:49:30.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-09-15T02:49:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:bucket 'host3' already exists 2024-09-15T02:49:31.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-09-15T02:49:31.603 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-09-15T02:49:31.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-09-15T02:49:32.181 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-09-15T02:49:32.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-09-15T02:49:32.790 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-09-15T02:49:32.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-09-15T02:49:33.327 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-09-15T02:49:33.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-09-15T02:49:33.887 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-09-15T02:49:33.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-09-15T02:49:34.425 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-09-15T02:49:34.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-09-15T02:49:34.975 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-09-15T02:49:34.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-09-15T02:49:35.521 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-09-15T02:49:35.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-09-15T02:49:36.048 INFO:tasks.workunit.client.0.smithi137.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-09-15T02:49:36.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-09-15T02:49:36.530 INFO:tasks.workunit.client.0.smithi137.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-09-15T02:49:36.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-09-15T02:49:37.032 INFO:tasks.workunit.client.0.smithi137.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-09-15T02:49:37.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-09-15T02:49:37.638 INFO:tasks.workunit.client.0.smithi137.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-09-15T02:49:37.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-09-15T02:49:38.122 INFO:tasks.workunit.client.0.smithi137.stderr:rule myrule already exists 2024-09-15T02:49:38.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-09-15T02:49:38.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-09-15T02:49:38.614 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'foo' already exists 2024-09-15T02:49:38.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:49:39.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-15T02:49:39.657 INFO:tasks.workunit.client.0.smithi137.stdout:hints = 1 2024-09-15T02:49:39.657 INFO:tasks.workunit.client.0.smithi137.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-15T02:49:39.657 INFO:tasks.workunit.client.0.smithi137.stdout:Object prefix: benchmark_data_smithi137_53535 2024-09-15T02:49:39.657 INFO:tasks.workunit.client.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T02:49:39.657 INFO:tasks.workunit.client.0.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-15T02:49:40.657 INFO:tasks.workunit.client.0.smithi137.stdout: 1 16 51 35 0.136711 0.136719 0.238171 0.438736 2024-09-15T02:49:41.657 INFO:tasks.workunit.client.0.smithi137.stdout: 2 16 100 84 0.164045 0.191406 0.240224 0.343345 2024-09-15T02:49:42.657 INFO:tasks.workunit.client.0.smithi137.stdout: 3 16 179 163 0.212216 0.308594 0.224269 0.289044 2024-09-15T02:49:43.658 INFO:tasks.workunit.client.0.smithi137.stdout: 4 16 228 212 0.207008 0.191406 0.256231 0.288027 2024-09-15T02:49:44.657 INFO:tasks.workunit.client.0.smithi137.stdout: 5 16 293 277 0.216382 0.253906 0.243654 0.277302 2024-09-15T02:49:45.658 INFO:tasks.workunit.client.0.smithi137.stdout: 6 15 360 345 0.224585 0.265625 0.28395 0.269477 2024-09-15T02:49:46.658 INFO:tasks.workunit.client.0.smithi137.stdout: 7 16 435 419 0.23379 0.289062 0.270206 0.265561 2024-09-15T02:49:47.658 INFO:tasks.workunit.client.0.smithi137.stdout: 8 16 499 483 0.235813 0.25 0.235453 0.261527 2024-09-15T02:49:48.658 INFO:tasks.workunit.client.0.smithi137.stdout: 9 16 548 532 0.230876 0.191406 0.490162 0.259342 2024-09-15T02:49:49.658 INFO:tasks.workunit.client.0.smithi137.stdout: 10 16 612 596 0.232785 0.25 0.260022 0.263506 2024-09-15T02:49:50.658 INFO:tasks.workunit.client.0.smithi137.stdout:Total time run: 10.1787 2024-09-15T02:49:50.658 INFO:tasks.workunit.client.0.smithi137.stdout:Total writes made: 612 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Write size: 4096 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Object size: 4096 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Bandwidth (MB/sec): 0.234867 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Stddev Bandwidth: 0.0531212 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Max bandwidth (MB/sec): 0.308594 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Min bandwidth (MB/sec): 0.136719 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Average IOPS: 60 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Stddev IOPS: 13.599 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Max IOPS: 79 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Min IOPS: 35 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Average Latency(s): 0.263492 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Stddev Latency(s): 0.0870015 2024-09-15T02:49:50.659 INFO:tasks.workunit.client.0.smithi137.stdout:Max latency(s): 0.757256 2024-09-15T02:49:50.660 INFO:tasks.workunit.client.0.smithi137.stdout:Min latency(s): 0.194592 2024-09-15T02:49:50.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-09-15T02:50:00.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-09-15T02:50:00.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:50:00.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:50:01.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:50:01.133 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:50:01.133 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T02:50:01.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:50:01.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:01.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:50:01.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-15T02:50:01.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-15T02:50:01.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-15T02:50:01.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:01.239 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:50:01.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672983 2024-09-15T02:50:01.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672983 2024-09-15T02:50:01.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672983' 2024-09-15T02:50:01.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:01.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:50:01.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-15T02:50:01.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-15T02:50:01.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672983 2-64424509455' 2024-09-15T02:50:01.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:01.460 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-15T02:50:01.460 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:01.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:50:01.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-15T02:50:01.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:01.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-15T02:50:01.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-15T02:50:01.462 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836507 2024-09-15T02:50:01.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:50:01.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-15T02:50:01.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:50:02.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:50:02.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:50:03.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-15T02:50:03.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:03.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672983 2024-09-15T02:50:03.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:03.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:50:03.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672983 2024-09-15T02:50:03.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:03.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672983 2024-09-15T02:50:03.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-15T02:50:03.254 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672983 2024-09-15T02:50:03.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:50:03.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-15T02:50:03.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:03.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-15T02:50:03.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:03.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:50:03.634 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-15T02:50:03.634 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:03.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-15T02:50:03.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-15T02:50:03.635 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509455 2024-09-15T02:50:03.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:50:03.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-15T02:50:03.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-09-15T02:50:03.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-09-15T02:50:04.411 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.375ms); Slow OSD heartbeats on front (longest 0.391ms); 1 pool(s) do not have an application enabled 2024-09-15T02:50:04.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-09-15T02:50:04.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-09-15T02:50:04.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.375ms); Slow OSD heartbeats on front (longest 0.391ms); 1 pool(s) do not have an application enabled 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.375ms) 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.375 msec 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.369 msec 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.340 msec 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.325 msec 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.315 msec 2024-09-15T02:50:04.892 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.305 msec 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.391ms) 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.391 msec 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.379 msec 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.363 msec 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.334 msec 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.320 msec 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.317 msec 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: application not enabled on pool 'foo' 2024-09-15T02:50:04.893 INFO:tasks.workunit.client.0.smithi137.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-15T02:50:04.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-09-15T02:50:04.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-09-15T02:50:04.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-09-15T02:50:04.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-09-15T02:50:04.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:04.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:05.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:05.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:50:05.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:50:05.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:50:05.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:50:05.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:50:05.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:50:05.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:05.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:50:05.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:50:05.018 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:05.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:50:05.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:50:05.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-15T02:50:05.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:50:05.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:05.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:50:05.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41201 2024-09-15T02:50:05.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:50:05.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:50:05.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T02:50:05.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/network-ping 0 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:05.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:05.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:50:05.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:50:05.043 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:50:05.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:50:05.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:50:05.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:50:05.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:05.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:50:05.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:50:05.045 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:05.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:50:05.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T02:50:05.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-15T02:50:05.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:50:05.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:05.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41201 2024-09-15T02:50:05.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41201 2024-09-15T02:50:05.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:50:05.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:50:05.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T02:50:05.050 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:50:05.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:50:05.077 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-09-15T02:50:05.077 DEBUG:teuthology.orchestra.run.smithi137:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-09-15T02:50:05.139 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:50:05.145 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:50:05.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/ok-to-stop 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-09-15T02:50:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-09-15T02:50:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-09-15T02:50:05.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-09-15T02:50:05.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none ' 2024-09-15T02:50:05.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none ' 2024-09-15T02:50:05.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none ' 2024-09-15T02:50:05.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-09-15T02:50:05.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T02:50:05.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-09-15T02:50:05.150 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_0_osd 2024-09-15T02:50:05.150 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_1_mon_checks 2024-09-15T02:50:05.150 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_2_mons_checks 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_3_mons_checks 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_4_mons_checks 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_5_mons_checks' 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:05.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:05.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:05.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:05.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:05.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:05.152 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:50:05.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:50:05.152 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:50:05.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:50:05.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:50:05.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:50:05.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:05.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:50:05.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:50:05.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:05.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:50:05.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:50:05.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:50:05.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:50:05.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:05.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:05.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:50:05.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:50:05.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:50:05.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-15T02:50:05.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:50:05.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:05.158 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:05.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53903 2024-09-15T02:50:05.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:50:05.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:50:05.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:50:05.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:50:05.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-15T02:50:05.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-09-15T02:50:05.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-15T02:50:05.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:05.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-15T02:50:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:50:05.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:50:05.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:50:05.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:50:05.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:50:05.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:50:05.814 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:50:05.814 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:50:05.815 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:50:05.815 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:50:05.815 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:05.815 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:05.815 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:50:05.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:50:05.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get fsid 2024-09-15T02:50:05.889 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:05.890 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:50:05.891 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:50:05.891 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get mon_host 2024-09-15T02:50:05.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-09-15T02:50:05.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-15T02:50:05.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:50:05.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:50:05.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:50:05.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-15T02:50:05.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:06.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:50:06.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:50:06.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-15T02:50:06.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:50:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-15T02:50:06.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:50:06.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c0e0519-bc83-42a1-8683-73c13e79f761 2024-09-15T02:50:06.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4c0e0519-bc83-42a1-8683-73c13e79f761' 2024-09-15T02:50:06.820 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 4c0e0519-bc83-42a1-8683-73c13e79f761 2024-09-15T02:50:06.820 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:50:06.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeS+Zm4nFqMRAAF63cq4LWiSPdWIsV1KDkCg== 2024-09-15T02:50:06.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeS+Zm4nFqMRAAF63cq4LWiSPdWIsV1KDkCg=="}' 2024-09-15T02:50:06.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c0e0519-bc83-42a1-8683-73c13e79f761 -i td/ok-to-stop/0/new.json 2024-09-15T02:50:07.093 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:50:07.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-15T02:50:07.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeS+Zm4nFqMRAAF63cq4LWiSPdWIsV1KDkCg== --osd-uuid 4c0e0519-bc83-42a1-8683-73c13e79f761 2024-09-15T02:50:07.126 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:07.125+0000 7f01ddf24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:07.139 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:07.138+0000 7f01ddf24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:07.140 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:07.139+0000 7f01ddf24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:07.140 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:07.140+0000 7f01ddf24440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-15T02:50:07.140 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:07.140+0000 7f01ddf24440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-15T02:50:07.140 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:07.140+0000 7f01ddf24440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-15T02:50:07.179 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:07.179+0000 7f01ddf24440 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-15T02:50:12.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-15T02:50:12.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:50:12.587 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:50:12.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:50:12.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:50:13.067 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:50:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:50:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:50:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:50:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:50:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:50:13.088 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:13.087+0000 7f2b40a10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:13.090 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:13.090+0000 7f2b40a10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:13.091 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:13.091+0000 7f2b40a10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:13.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:50:13.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:50:13.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:50:13.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:50:13.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:50:13.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:13.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:50:13.415 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:50:13.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:13.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:13.606 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:13.606+0000 7f2b40a10440 -1 Falling back to public interface 2024-09-15T02:50:13.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:14.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:14.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:14.786 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:50:14.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:50:14.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:14.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:15.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:16.133 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:16.132+0000 7f2b40a10440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:50:16.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:16.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:16.136 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:50:16.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:50:16.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:16.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:16.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:17.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:17.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:17.500 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:50:17.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:50:17.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:17.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:17.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:18.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:18.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:18.979 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:50:18.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:50:18.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:18.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:19.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:20.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:20.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:20.366 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:50:20.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:50:20.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:20.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:20.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:21.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:21.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:21.820 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:50:21.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:50:21.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:21.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:22.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:23.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:23.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:23.199 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:50:23.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:50:23.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:23.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:23.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:24.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:24.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:24.583 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:50:24.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:50:24.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:24.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:24.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:25.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:25.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:25.968 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:50:25.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:50:25.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:25.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:26.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:27.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:27.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:27.326 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:50:27.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:50:27.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:27.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:27.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:28.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:28.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:28.783 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:50:28.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:50:28.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:28.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:29.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:30.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:30.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:30.235 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:50:30.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:50:30.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:30.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:31.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:32.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:32.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:32.293 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:50:32.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:50:32.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:32.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:33.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:34.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:34.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:34.184 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:50:34.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:50:34.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:34.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:34.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:35.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:35.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:35.903 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:50:35.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:50:35.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:35.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:36.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:37.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:37.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:37.575 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:50:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:50:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:37.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:38.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:38.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:38.933 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:50:38.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:50:38.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:38.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:50:39.329 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/54994,v1:127.0.0.1:6803/54994] [v2:127.0.0.1:6804/54994,v1:127.0.0.1:6805/54994] exists,up 4c0e0519-bc83-42a1-8683-73c13e79f761 2024-09-15T02:50:39.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:50:39.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:50:39.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:39.332 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:39.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:39.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-15T02:50:39.350 INFO:tasks.workunit.client.0.smithi137.stdout:starting mds.a at 2024-09-15T02:50:39.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-09-15T02:50:39.793 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'meta' already exists 2024-09-15T02:50:39.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-09-15T02:50:40.617 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'data' already exists 2024-09-15T02:50:40.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-09-15T02:50:41.572 INFO:tasks.workunit.client.0.smithi137.stderr:filesystem 'myfs' already exists 2024-09-15T02:50:41.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-09-15T02:50:46.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-09-15T02:50:46.899 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-15T02:50:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-09-15T02:50:47.208 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-15T02:50:47.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-09-15T02:50:47.671 INFO:tasks.workunit.client.0.smithi137.stderr:should be safe to stop [dne] 2024-09-15T02:50:47.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-09-15T02:50:47.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-15T02:50:47.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:47.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-15T02:50:47.702 INFO:tasks.workunit.client.0.smithi137.stdout:starting mds.b at 2024-09-15T02:50:47.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-09-15T02:50:52.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-09-15T02:50:53.180 INFO:tasks.workunit.client.0.smithi137.stderr:should be safe to stop [a] 2024-09-15T02:50:53.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-09-15T02:50:53.666 INFO:tasks.workunit.client.0.smithi137.stderr:should be safe to stop [b] 2024-09-15T02:50:53.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-09-15T02:50:53.980 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-15T02:50:53.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-09-15T02:50:54.457 INFO:tasks.workunit.client.0.smithi137.stderr:should be safe to stop [a,dne1,dne2] 2024-09-15T02:50:54.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-09-15T02:50:54.925 INFO:tasks.workunit.client.0.smithi137.stderr:should be safe to stop [b,dne] 2024-09-15T02:50:54.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-09-15T02:50:55.237 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-15T02:50:55.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-09-15T02:50:55.705 INFO:tasks.workunit.client.0.smithi137.stderr:should be safe to stop [dne1,dne2] 2024-09-15T02:50:55.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-09-15T02:50:55.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:55.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:55.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:55.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:55.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-15T02:50:55.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:55.822 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:55.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:55.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-15T02:50:55.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:50:55.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:50:55.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:50:55.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:55.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:55.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:55.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:55.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:55.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:55.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:50:55.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:50:55.935 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:50:55.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:50:55.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:50:55.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:50:55.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:55.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:50:55.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:50:55.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:55.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:50:55.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:50:55.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:50:55.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:50:55.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:55.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:55.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:50:55.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:50:55.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:50:55.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-15T02:50:55.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:55.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:55.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:55.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:50:55.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:50:55.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:50:55.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:50:55.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:50:55.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:50:55.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:55.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:50:55.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:50:55.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:50:55.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:50:55.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:50:55.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:50:55.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:50:55.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:55.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:55.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:50:55.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:50:55.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:50:55.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-15T02:50:55.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:50:55.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:55.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:55.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53903 2024-09-15T02:50:55.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:50:55.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-15T02:50:55.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-15T02:50:56.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:50:56.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:56.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:56.301 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:56.302 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:56.302 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:56.302 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:56.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-15T02:50:56.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:50:56.536 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:50:56.536 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:50:56.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:50:56.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:50:56.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:50:56.537 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:50:56.537 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:50:56.537 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:50:56.538 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:50:56.538 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:56.538 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:56.538 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:50:56.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:50:56.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get fsid 2024-09-15T02:50:56.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:50:56.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:50:56.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:50:56.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:50:56.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get mon_host 2024-09-15T02:50:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-09-15T02:50:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-15T02:50:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:50:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:50:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:50:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-15T02:50:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:50:57.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:50:57.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:50:57.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:57.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:50:57.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:50:57.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:50:57.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-15T02:50:57.550 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:50:57.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1bb553a1-3ab1-4ad0-a48d-13b0a877e742 2024-09-15T02:50:57.550 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 1bb553a1-3ab1-4ad0-a48d-13b0a877e742 2024-09-15T02:50:57.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1bb553a1-3ab1-4ad0-a48d-13b0a877e742' 2024-09-15T02:50:57.551 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:50:57.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRS+ZmiERUIRAAMS8ILK5hphZ5Pe+Fcj+3CQ== 2024-09-15T02:50:57.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRS+ZmiERUIRAAMS8ILK5hphZ5Pe+Fcj+3CQ=="}' 2024-09-15T02:50:57.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1bb553a1-3ab1-4ad0-a48d-13b0a877e742 -i td/ok-to-stop/0/new.json 2024-09-15T02:50:57.857 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:50:57.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-15T02:50:57.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRS+ZmiERUIRAAMS8ILK5hphZ5Pe+Fcj+3CQ== --osd-uuid 1bb553a1-3ab1-4ad0-a48d-13b0a877e742 2024-09-15T02:50:57.889 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:57.888+0000 7f5093822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:57.903 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:57.902+0000 7f5093822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:57.904 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:57.903+0000 7f5093822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:57.905 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:57.903+0000 7f5093822440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-15T02:50:57.905 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:57.903+0000 7f5093822440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-15T02:50:57.905 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:57.903+0000 7f5093822440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-15T02:50:57.944 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:50:57.942+0000 7f5093822440 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-15T02:51:03.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-15T02:51:03.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:51:03.274 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:51:03.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:51:03.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:51:03.756 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:51:03.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:51:03.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:51:03.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:51:03.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:51:03.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:51:03.777 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:03.776+0000 7f55a4b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:03.778 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:03.778+0000 7f55a4b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:03.780 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:03.779+0000 7f55a4b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:04.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:51:04.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:51:04.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:51:04.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:51:04.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:51:04.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:04.103 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:51:04.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:51:04.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:04.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:04.293 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:04.293+0000 7f55a4b23440 -1 Falling back to public interface 2024-09-15T02:51:04.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:05.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:05.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:05.474 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:51:05.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:51:05.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:05.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:05.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:06.771 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:06.770+0000 7f55a4b23440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:51:06.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:06.824 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:51:06.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:06.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:51:06.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:06.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:07.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:08.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:08.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:08.364 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:51:08.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:51:08.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:08.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:08.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:09.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:09.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:09.793 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:51:09.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:51:09.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:09.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:10.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:11.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:11.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:11.182 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:51:11.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:51:11.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:11.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:11.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:12.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:12.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:12.671 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:51:12.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:51:12.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:12.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:13.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:14.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:14.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:14.069 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:51:14.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:51:14.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:14.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:14.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:15.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:15.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:15.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:51:15.484 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:51:15.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:15.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:15.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:16.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:16.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:16.871 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:51:16.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:51:16.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:16.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:17.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:18.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:18.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:18.268 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:51:18.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:51:18.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:18.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:19.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:20.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:20.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:20.106 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:51:20.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:51:20.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:20.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:20.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:21.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:21.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:21.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:51:21.605 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:51:21.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:21.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:22.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:23.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:23.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:23.285 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:51:23.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:51:23.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:23.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:24.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:25.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:25.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:25.013 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:51:25.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:51:25.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:25.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:25.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:26.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:26.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:26.522 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:51:26.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:51:26.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:26.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:27.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:28.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:28.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:28.286 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:51:28.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:51:28.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:28.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:29.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:30.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:30.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:30.087 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:51:30.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:51:30.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:30.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:51:30.440 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/57417,v1:127.0.0.1:6803/57417] [v2:127.0.0.1:6804/57417,v1:127.0.0.1:6805/57417] exists,up 1bb553a1-3ab1-4ad0-a48d-13b0a877e742 2024-09-15T02:51:30.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:51:30.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:51:30.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:51:30.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-09-15T02:51:30.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-15T02:51:30.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-09-15T02:51:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:51:30.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:51:30.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-15T02:51:30.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:51:30.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:51:30.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:51:30.446 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:51:30.446 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:51:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0266812d-8b7c-443f-839c-d14e98eaf74d 2024-09-15T02:51:30.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0266812d-8b7c-443f-839c-d14e98eaf74d' 2024-09-15T02:51:30.449 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 0266812d-8b7c-443f-839c-d14e98eaf74d 2024-09-15T02:51:30.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:51:30.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyS+ZmOBwhGxAANJFSKQ20dHwVhfyW9ddQbg== 2024-09-15T02:51:30.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyS+ZmOBwhGxAANJFSKQ20dHwVhfyW9ddQbg=="}' 2024-09-15T02:51:30.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0266812d-8b7c-443f-839c-d14e98eaf74d -i td/ok-to-stop/1/new.json 2024-09-15T02:51:30.943 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:51:30.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-09-15T02:51:30.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyS+ZmOBwhGxAANJFSKQ20dHwVhfyW9ddQbg== --osd-uuid 0266812d-8b7c-443f-839c-d14e98eaf74d 2024-09-15T02:51:30.978 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:30.976+0000 7fc855723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:30.979 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:30.978+0000 7fc855723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:30.980 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:30.979+0000 7fc855723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:30.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:30.979+0000 7fc855723440 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-15T02:51:30.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:30.979+0000 7fc855723440 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-15T02:51:30.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:30.979+0000 7fc855723440 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-15T02:51:31.058 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:31.057+0000 7fc855723440 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-09-15T02:51:37.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-09-15T02:51:37.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:51:37.133 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T02:51:37.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:51:37.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:51:37.608 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T02:51:37.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:51:37.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:51:37.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:51:37.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:51:37.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:51:37.629 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:37.627+0000 7ff0dae45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:37.630 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:37.629+0000 7ff0dae45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:37.631 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:37.630+0000 7ff0dae45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:37.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:51:37.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:51:37.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:51:37.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:51:37.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:51:37.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:37.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:51:37.997 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:51:37.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:37.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:38.146 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:38.145+0000 7ff0dae45440 -1 Falling back to public interface 2024-09-15T02:51:38.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:39.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:39.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:39.344 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:51:39.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:51:39.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:39.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:39.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:40.636 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:51:40.635+0000 7ff0dae45440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:51:40.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:40.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:40.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:51:40.701 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:51:40.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:40.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:41.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:42.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:42.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:42.212 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:51:42.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:51:42.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:42.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:42.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:43.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:43.569 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:51:43.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:43.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:51:43.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:43.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:44.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:45.063 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:51:45.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:51:45.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:45.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:45.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:46.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:46.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:46.523 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:51:46.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:51:46.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:46.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:46.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:47.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:47.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:47.913 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:51:47.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:51:47.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:47.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:48.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:49.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:49.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:49.449 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:51:49.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:51:49.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:49.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:49.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:50.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:50.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:50.926 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:51:50.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:51:50.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:50.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:51.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:52.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:52.283 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:51:52.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:52.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:51:52.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:52.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:52.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:53.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:53.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:53.728 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:51:53.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:51:53.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:53.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:54.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:55.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:55.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:55.110 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:51:55.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:51:55.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:55.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:55.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:56.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:56.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:56.522 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:51:56.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:51:56.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:56.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:57.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:58.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:58.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:58.251 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:51:58.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:51:58.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:58.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:51:58.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:59.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:59.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:59.960 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:51:59.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:51:59.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:59.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:01.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:02.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:02.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:02.009 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:52:02.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:52:02.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:02.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:02.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:03.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:03.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:03.796 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:52:03.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:52:03.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:03.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:04.149 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/59171,v1:127.0.0.1:6811/59171] [v2:127.0.0.1:6812/59171,v1:127.0.0.1:6813/59171] exists,up 0266812d-8b7c-443f-839c-d14e98eaf74d 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-09-15T02:52:04.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:52:04.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:52:04.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:52:04.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:52:04.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-09-15T02:52:04.158 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:52:04.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c6964a3-1c33-4973-847b-536b635aa0b6 2024-09-15T02:52:04.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0c6964a3-1c33-4973-847b-536b635aa0b6' 2024-09-15T02:52:04.158 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 0c6964a3-1c33-4973-847b-536b635aa0b6 2024-09-15T02:52:04.158 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:52:04.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUS+ZmLdrYCRAAQRkU6DreSElpG7+dUigWpg== 2024-09-15T02:52:04.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUS+ZmLdrYCRAAQRkU6DreSElpG7+dUigWpg=="}' 2024-09-15T02:52:04.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c6964a3-1c33-4973-847b-536b635aa0b6 -i td/ok-to-stop/2/new.json 2024-09-15T02:52:04.623 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:52:04.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-09-15T02:52:04.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUS+ZmLdrYCRAAQRkU6DreSElpG7+dUigWpg== --osd-uuid 0c6964a3-1c33-4973-847b-536b635aa0b6 2024-09-15T02:52:04.656 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:04.655+0000 7f1e3ae0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:04.657 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:04.657+0000 7f1e3ae0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:04.658 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:04.658+0000 7f1e3ae0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:04.659 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:04.658+0000 7f1e3ae0a440 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-15T02:52:04.659 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:04.658+0000 7f1e3ae0a440 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-15T02:52:04.659 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:04.658+0000 7f1e3ae0a440 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-15T02:52:04.696 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:04.696+0000 7f1e3ae0a440 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-09-15T02:52:10.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-09-15T02:52:10.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:52:10.909 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T02:52:10.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:52:10.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:52:11.391 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T02:52:11.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:52:11.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:52:11.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:52:11.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:52:11.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:52:11.412 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:11.412+0000 7f6fcba0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:11.414 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:11.414+0000 7f6fcba0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:11.415 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:11.415+0000 7f6fcba0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:11.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:52:11.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:52:11.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:52:11.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:52:11.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:52:11.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:11.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:52:11.819 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:52:11.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:11.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:11.944 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:11.944+0000 7f6fcba0a440 -1 Falling back to public interface 2024-09-15T02:52:12.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:13.167 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:52:13.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:52:13.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:13.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:13.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:14.434 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:14.434+0000 7f6fcba0a440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:52:14.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:14.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:14.572 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:52:14.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:52:14.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:14.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:16.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:16.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:16.118 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:52:16.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:52:16.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:16.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:16.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:17.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:17.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:17.508 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:52:17.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:52:17.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:17.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:17.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:18.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:18.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:18.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:52:18.891 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:52:18.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:18.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:19.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:20.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:20.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:20.387 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:52:20.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:52:20.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:20.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:20.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:21.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:21.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:21.752 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:52:21.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:52:21.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:21.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:22.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:23.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:23.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:23.193 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:52:23.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:52:23.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:23.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:23.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:24.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:24.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:24.578 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:52:24.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:52:24.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:24.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:24.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:25.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:25.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:25.961 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:52:25.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:52:25.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:25.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:26.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:27.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:27.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:27.459 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:52:27.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:52:27.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:27.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:28.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:29.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:29.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:29.228 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:52:29.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:52:29.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:29.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:29.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:30.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:30.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:30.889 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:52:30.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:52:30.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:30.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:31.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:32.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:32.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:32.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:52:32.439 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:52:32.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:32.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:33.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:34.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:34.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:34.261 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:52:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:52:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:35.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:36.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:36.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:36.089 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:52:36.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:52:36.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:36.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:36.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:37.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:37.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:37.695 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:52:37.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:52:37.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:37.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:52:38.055 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/60925,v1:127.0.0.1:6819/60925] [v2:127.0.0.1:6820/60925,v1:127.0.0.1:6821/60925] exists,up 0c6964a3-1c33-4973-847b-536b635aa0b6 2024-09-15T02:52:38.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:52:38.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:52:38.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:52:38.059 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:52:38.061 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:52:38.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:52:38.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:52:38.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:52:38.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-09-15T02:52:38.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:52:38.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce08a39b-aca4-44ec-8e74-4b1365940c03 2024-09-15T02:52:38.063 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 ce08a39b-aca4-44ec-8e74-4b1365940c03 2024-09-15T02:52:38.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ce08a39b-aca4-44ec-8e74-4b1365940c03' 2024-09-15T02:52:38.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:52:38.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2S+ZmF984BBAAH/UvFKES5bleCmP+m+jbQw== 2024-09-15T02:52:38.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2S+ZmF984BBAAH/UvFKES5bleCmP+m+jbQw=="}' 2024-09-15T02:52:38.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce08a39b-aca4-44ec-8e74-4b1365940c03 -i td/ok-to-stop/3/new.json 2024-09-15T02:52:38.538 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:52:38.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-09-15T02:52:38.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2S+ZmF984BBAAH/UvFKES5bleCmP+m+jbQw== --osd-uuid ce08a39b-aca4-44ec-8e74-4b1365940c03 2024-09-15T02:52:38.572 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:38.570+0000 7f304e966440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:38.573 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:38.572+0000 7f304e966440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:38.574 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:38.573+0000 7f304e966440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:38.575 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:38.573+0000 7f304e966440 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-15T02:52:38.575 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:38.573+0000 7f304e966440 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-15T02:52:38.575 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:38.573+0000 7f304e966440 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-15T02:52:38.620 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:38.619+0000 7f304e966440 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-09-15T02:52:44.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-09-15T02:52:44.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:52:44.755 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-09-15T02:52:44.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:52:44.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:52:45.238 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-09-15T02:52:45.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:52:45.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:52:45.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:52:45.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:52:45.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:52:45.259 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:45.257+0000 7f02ed151440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:45.260 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:45.259+0000 7f02ed151440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:45.261 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:45.260+0000 7f02ed151440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:45.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:52:45.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:52:45.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:52:45.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:52:45.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:52:45.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:45.585 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:52:45.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:52:45.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:45.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:45.776 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:45.775+0000 7f02ed151440 -1 Falling back to public interface 2024-09-15T02:52:45.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:46.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:46.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:46.935 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:52:46.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:52:46.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:46.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:47.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:48.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:48.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:48.314 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:52:48.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:52:48.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:48.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:48.318 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:52:48.317+0000 7f02ed151440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:52:48.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:49.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:49.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:49.763 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:52:49.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:52:49.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:49.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:50.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:51.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:51.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:51.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:52:51.258 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:52:51.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:51.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:51.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:52.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:52.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:52.640 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:52:52.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:52:52.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:52.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:53.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:54.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:54.073 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:52:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:52:54.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:54.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:54.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:55.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:55.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:55.471 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:52:55.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:52:55.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:55.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:55.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:56.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:56.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:56.950 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:52:56.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:52:56.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:56.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:57.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:58.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:58.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:58.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:52:58.358 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:52:58.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:58.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:52:58.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:59.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:59.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:59.738 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:52:59.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:52:59.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:59.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:00.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:01.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:01.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:01.320 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:53:01.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:53:01.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:01.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:02.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:03.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:03.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:03.147 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:53:03.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:53:03.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:03.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:03.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:04.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:04.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:04.646 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:53:04.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:53:04.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:04.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:05.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:06.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:06.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:06.411 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:53:06.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:53:06.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:06.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:07.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:08.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:08.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:08.228 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:53:08.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:53:08.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:08.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:08.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:09.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:09.931 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:53:09.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:09.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:53:09.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:09.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:10.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:11.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:11.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:11.789 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:53:11.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:53:11.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:11.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:12.139 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/62679,v1:127.0.0.1:6827/62679] [v2:127.0.0.1:6828/62679,v1:127.0.0.1:6829/62679] exists,up ce08a39b-aca4-44ec-8e74-4b1365940c03 2024-09-15T02:53:12.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:53:12.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:53:12.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:53:12.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-15T02:53:12.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-09-15T02:53:13.363 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'ec' already exists 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:53:13.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:53:13.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:53:13.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:53:13.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:53:13.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:53:13.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:53:13.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:53:13.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:53:13.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:53:14.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:53:14.003 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:53:14.003 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-09-15T02:53:14.003 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-09-15T02:53:14.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:53:14.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:14.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:53:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-15T02:53:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-15T02:53:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T02:53:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:14.210 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:53:14.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-15T02:53:14.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-15T02:53:14.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-09-15T02:53:14.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:14.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:53:14.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-15T02:53:14.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-15T02:53:14.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509453' 2024-09-15T02:53:14.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:14.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:53:14.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345927 2024-09-15T02:53:14.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345927 2024-09-15T02:53:14.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509453 3-85899345927' 2024-09-15T02:53:14.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:14.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-15T02:53:14.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:14.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:53:14.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-15T02:53:14.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:14.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-15T02:53:14.859 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836509 2024-09-15T02:53:14.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T02:53:14.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:53:15.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-15T02:53:15.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:53:16.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:53:16.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:53:16.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T02:53:16.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:16.574 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-15T02:53:16.574 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:16.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:53:16.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-15T02:53:16.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:16.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-15T02:53:16.576 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672981 2024-09-15T02:53:16.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-15T02:53:16.577 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:53:16.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-15T02:53:16.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:16.925 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-15T02:53:16.925 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:16.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:53:16.928 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-15T02:53:16.928 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:16.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-15T02:53:16.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T02:53:16.929 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509453 2024-09-15T02:53:16.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:53:17.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-15T02:53:17.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:17.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345927 2024-09-15T02:53:17.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:17.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:53:17.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345927 2024-09-15T02:53:17.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:17.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345927 2024-09-15T02:53:17.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-09-15T02:53:17.276 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 85899345927 2024-09-15T02:53:17.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:53:17.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345928 -lt 85899345927 2024-09-15T02:53:17.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:53:17.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:17.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:18.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-15T02:53:18.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:53:18.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:53:18.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:53:18.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:53:18.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:53:18.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:53:18.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:53:18.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=15 2024-09-15T02:53:18.441 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:53:18.442 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:18.442 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:18.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 15 = 32 2024-09-15T02:53:18.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 15 '!=' -1 2024-09-15T02:53:18.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T02:53:18.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=15 2024-09-15T02:53:18.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:53:18.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:53:19.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:53:19.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2024-09-15T02:53:19.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:53:19.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:19.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:19.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2024-09-15T02:53:19.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' 15 2024-09-15T02:53:19.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T02:53:19.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=19 2024-09-15T02:53:19.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:53:19.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:53:19.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:53:19.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:53:19.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:53:19.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:53:19.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:53:19.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:53:19.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:53:19.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:53:20.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2024-09-15T02:53:20.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:53:20.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:20.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:20.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2024-09-15T02:53:20.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' 19 2024-09-15T02:53:20.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T02:53:20.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T02:53:20.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T02:53:20.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T02:53:20.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T02:53:20.777 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T02:53:20.777 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T02:53:21.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T02:53:21.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T02:53:21.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T02:53:21.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:53:21.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T02:53:21.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:53:21.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:53:21.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:53:21.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:53:21.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:53:21.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:53:21.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:53:21.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:53:21.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-15T02:53:21.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:53:21.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:21.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:22.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-15T02:53:22.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:53:22.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:53:22.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-09-15T02:53:22.887 INFO:tasks.workunit.client.0.smithi137.stderr:set pool 1 min_size to 3 2024-09-15T02:53:22.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-09-15T02:53:22.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:53:22.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:53:22.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:53:22.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:53:22.899 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:53:22.899 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:53:22.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:53:22.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:53:22.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:53:22.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:53:22.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:53:22.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:53:22.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:53:22.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:53:22.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:53:23.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:53:23.355 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:53:23.355 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-09-15T02:53:23.355 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-09-15T02:53:23.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:53:23.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:23.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:53:23.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836513 2024-09-15T02:53:23.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836513 2024-09-15T02:53:23.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513' 2024-09-15T02:53:23.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:23.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:53:23.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672984 2024-09-15T02:53:23.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672984 2024-09-15T02:53:23.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672984' 2024-09-15T02:53:23.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:23.601 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:53:23.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-15T02:53:23.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-15T02:53:23.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672984 2-64424509457' 2024-09-15T02:53:23.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:23.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:53:23.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345931 2024-09-15T02:53:23.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345931 2024-09-15T02:53:23.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672984 2-64424509457 3-85899345931' 2024-09-15T02:53:23.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:23.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836513 2024-09-15T02:53:23.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:23.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:53:23.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836513 2024-09-15T02:53:23.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:23.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836513 2024-09-15T02:53:23.847 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836513 2024-09-15T02:53:23.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-15T02:53:23.847 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:53:24.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836513 2024-09-15T02:53:24.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:53:25.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:53:25.197 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:53:25.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-15T02:53:25.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:25.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672984 2024-09-15T02:53:25.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:25.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:53:25.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672984 2024-09-15T02:53:25.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:25.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672984 2024-09-15T02:53:25.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-15T02:53:25.551 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672984 2024-09-15T02:53:25.551 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:53:25.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672984 2024-09-15T02:53:25.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:25.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-15T02:53:25.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:25.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:53:25.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-15T02:53:25.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:25.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-15T02:53:25.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T02:53:25.900 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509457 2024-09-15T02:53:25.900 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:53:26.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-15T02:53:26.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:26.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345931 2024-09-15T02:53:26.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:26.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:53:26.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345931 2024-09-15T02:53:26.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:26.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345931 2024-09-15T02:53:26.247 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 85899345931 2024-09-15T02:53:26.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-09-15T02:53:26.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:53:26.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345931 -lt 85899345931 2024-09-15T02:53:26.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:53:26.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:26.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:53:27.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:53:27.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-15T02:53:27.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:53:27.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:27.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:27.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-15T02:53:27.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:53:27.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:53:27.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-15T02:53:28.266 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:28.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-09-15T02:53:28.606 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:28.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-09-15T02:53:28.975 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:28.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-09-15T02:53:29.341 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:29.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-15T02:53:29.588 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:29.588 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-15T02:53:29.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-15T02:53:29.827 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:29.827 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-15T02:53:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-15T02:53:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-09-15T02:53:30.171 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:30.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-09-15T02:53:30.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-09-15T02:53:30.511 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:30.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-09-15T02:53:31.987 INFO:tasks.workunit.client.0.smithi137.stderr:set pool 1 min_size to 2 2024-09-15T02:53:31.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-09-15T02:53:32.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:53:32.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:53:32.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:53:32.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:53:32.001 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:53:32.001 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:53:32.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:53:32.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:53:32.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:53:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:53:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:53:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:53:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:53:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:53:32.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:53:32.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:53:32.566 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:53:32.566 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-09-15T02:53:32.566 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-09-15T02:53:32.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:53:32.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:32.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:53:32.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836517 2024-09-15T02:53:32.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836517 2024-09-15T02:53:32.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517' 2024-09-15T02:53:32.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:32.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:53:32.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672988 2024-09-15T02:53:32.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672988 2024-09-15T02:53:32.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672988' 2024-09-15T02:53:32.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:32.844 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:53:32.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509461 2024-09-15T02:53:32.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509461 2024-09-15T02:53:32.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672988 2-64424509461' 2024-09-15T02:53:32.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:32.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:53:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345935 2024-09-15T02:53:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345935 2024-09-15T02:53:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672988 2-64424509461 3-85899345935' 2024-09-15T02:53:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:33.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836517 2024-09-15T02:53:33.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:33.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:53:33.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836517 2024-09-15T02:53:33.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:33.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836517 2024-09-15T02:53:33.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-15T02:53:33.051 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836517 2024-09-15T02:53:33.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:53:33.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-15T02:53:33.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:33.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672988 2024-09-15T02:53:33.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:33.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:53:33.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672988 2024-09-15T02:53:33.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672988 2024-09-15T02:53:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-15T02:53:33.398 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672988 2024-09-15T02:53:33.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:53:33.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-15T02:53:33.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:33.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509461 2024-09-15T02:53:33.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:33.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:53:33.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509461 2024-09-15T02:53:33.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:33.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509461 2024-09-15T02:53:33.751 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509461 2024-09-15T02:53:33.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-15T02:53:33.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:53:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-15T02:53:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:34.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345935 2024-09-15T02:53:34.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:34.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:53:34.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345935 2024-09-15T02:53:34.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:34.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345935 2024-09-15T02:53:34.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345935' 2024-09-15T02:53:34.106 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 85899345935 2024-09-15T02:53:34.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:53:34.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345933 -lt 85899345935 2024-09-15T02:53:34.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:53:35.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:53:35.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:53:35.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345935 -lt 85899345935 2024-09-15T02:53:35.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:53:35.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:35.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:36.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-15T02:53:36.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:53:36.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:53:36.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:53:36.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:53:36.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:53:36.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:53:36.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:53:36.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-15T02:53:36.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:53:36.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:36.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-15T02:53:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:53:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:53:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-15T02:53:37.497 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:37.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-15T02:53:37.844 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:37.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-15T02:53:38.090 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:38.090 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-15T02:53:38.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-15T02:53:38.330 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:38.330 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-15T02:53:38.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-15T02:53:38.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-09-15T02:53:38.675 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:38.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-09-15T02:53:38.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-09-15T02:53:39.020 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:39.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-09-15T02:53:39.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-09-15T02:53:39.361 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:39.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-09-15T02:53:39.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-09-15T02:53:39.703 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:39.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-09-15T02:53:39.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:39.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:41.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:41.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-09-15T02:53:41.449 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already down. 2024-09-15T02:53:41.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-09-15T02:53:41.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-15T02:53:41.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-15T02:53:41.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-15T02:53:41.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-15T02:53:41.461 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:53:41.461 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:53:41.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:53:41.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:53:41.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:53:41.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:53:41.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-15T02:53:41.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-15T02:53:41.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-15T02:53:41.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:53:41.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:53:41.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:53:41.895 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T02:53:41.896 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-09-15T02:53:41.896 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-09-15T02:53:41.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:53:41.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:41.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:53:41.977 INFO:tasks.workunit.client.0.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-15T02:53:41.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T02:53:41.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T02:53:41.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T02:53:41.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:41.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:53:42.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672992 2024-09-15T02:53:42.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672992 2024-09-15T02:53:42.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672992' 2024-09-15T02:53:42.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:42.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:53:42.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509464 2024-09-15T02:53:42.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509464 2024-09-15T02:53:42.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672992 2-64424509464' 2024-09-15T02:53:42.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:53:42.186 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:53:42.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345939 2024-09-15T02:53:42.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345939 2024-09-15T02:53:42.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672992 2-64424509464 3-85899345939' 2024-09-15T02:53:42.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:42.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672992 2024-09-15T02:53:42.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:42.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:53:42.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672992 2024-09-15T02:53:42.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:42.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672992 2024-09-15T02:53:42.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-15T02:53:42.291 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672992 2024-09-15T02:53:42.291 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:53:42.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672990 -lt 42949672992 2024-09-15T02:53:42.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:53:43.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:53:43.634 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:53:44.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-15T02:53:44.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:44.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509464 2024-09-15T02:53:44.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:44.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:53:44.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509464 2024-09-15T02:53:44.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:44.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509464 2024-09-15T02:53:44.008 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509464 2024-09-15T02:53:44.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-15T02:53:44.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:53:44.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509465 -lt 64424509464 2024-09-15T02:53:44.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:53:44.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345939 2024-09-15T02:53:44.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:53:44.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:53:44.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345939 2024-09-15T02:53:44.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:53:44.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345939 2024-09-15T02:53:44.379 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 85899345939 2024-09-15T02:53:44.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-09-15T02:53:44.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:53:44.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-15T02:53:44.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-15T02:53:44.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:44.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:45.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-09-15T02:53:45.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-15T02:53:45.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-15T02:53:45.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-15T02:53:45.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-15T02:53:45.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-15T02:53:45.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-15T02:53:45.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-15T02:53:45.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-09-15T02:53:45.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-15T02:53:45.550 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:53:45.550 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:53:46.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-09-15T02:53:46.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-15T02:53:46.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-15T02:53:46.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-15T02:53:46.383 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-09-15T02:53:46.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-15T02:53:46.729 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:46.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-15T02:53:46.977 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:46.977 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-15T02:53:46.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-15T02:53:47.240 INFO:tasks.workunit.client.0.smithi137.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-15T02:53:47.240 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-15T02:53:47.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:47.244 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:47.244 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:47.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:47.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:47.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-15T02:53:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:53:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:53:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:47.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:47.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:47.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:47.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:53:47.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:53:47.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:53:47.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:53:47.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:53:47.361 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:53:47.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:47.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:53:47.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:53:47.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:47.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:53:47.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:53:47.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:53:47.387 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:53:47.387 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:47.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:47.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:53:47.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:53:47.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:53:47.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:47.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:53:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:53:47.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:53:47.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:53:47.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:53:47.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:53:47.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:47.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:53:47.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:53:47.394 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:47.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:53:47.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:53:47.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:53:47.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:53:47.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:47.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:47.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:53:47.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:53:47.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:53:47.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-15T02:53:47.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:53:47.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:47.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:47.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53903 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:53:47.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:53:47.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:53:47.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:53:47.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-15T02:53:47.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-15T02:53:47.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:53:47.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:53:47.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:53:47.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:53:47.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:47.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:47.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:53:47.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-15T02:53:48.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:53:48.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:53:48.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:48.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:53:48.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:53:48.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:53:48.038 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:53:48.038 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:53:48.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:53:48.039 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:48.039 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:48.039 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:48.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:53:48.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:48.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get fsid 2024-09-15T02:53:48.111 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:53:48.111 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:48.111 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:48.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get mon_host 2024-09-15T02:53:48.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-09-15T02:53:48.963 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a) after stopping [dne] 2024-09-15T02:53:48.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-09-15T02:53:49.131 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-09-15T02:53:49.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-09-15T02:53:49.277 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-15T02:53:49.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-09-15T02:53:49.427 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-15T02:53:49.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-09-15T02:53:49.616 INFO:tasks.workunit.client.0.smithi137.stderr:mon.dne does not exist 2024-09-15T02:53:49.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:49.626 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:49.626 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:49.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:49.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:49.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:49.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:49.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-15T02:53:49.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:53:49.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:53:49.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:49.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:49.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:49.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:49.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:49.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:49.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:49.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:53:49.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:53:49.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:53:49.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:53:49.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:53:49.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:53:49.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:49.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:53:49.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:53:49.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:49.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:53:49.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:53:49.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:53:49.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:53:49.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:49.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:49.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:53:49.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:53:49.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:53:49.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-15T02:53:49.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-15T02:53:49.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:49.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:49.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:49.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:53:49.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:53:49.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:53:49.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:53:49.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:53:49.747 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:53:49.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:49.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:53:49.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:53:49.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:49.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:53:49.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:53:49.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:53:49.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:53:49.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:49.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:49.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:53:49.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:53:49.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:53:49.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-15T02:53:49.753 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:53:49.753 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:49.753 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:49.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53903 2024-09-15T02:53:49.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:53:49.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-15T02:53:49.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-15T02:53:50.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:53:50.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:53:50.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:53:50.135 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:53:50.135 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:50.135 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:50.136 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:53:50.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-15T02:53:50.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:53:50.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:53:50.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:50.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:53:50.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:53:50.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:53:50.379 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:53:50.379 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:53:50.379 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:53:50.379 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:50.380 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:50.380 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:50.380 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:53:50.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:50.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get fsid 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:50.454 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:50.455 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:53:50.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:50.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get mon_host 2024-09-15T02:53:50.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-15T02:53:50.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:53:50.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:53:50.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T02:53:50.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:53:50.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-15T02:53:50.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-15T02:53:50.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:53:50.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:53:50.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:53:50.856 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:53:50.856 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:50.857 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:50.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:53:50.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-15T02:53:51.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:53:51.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T02:53:51.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:51.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:53:51.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:53:51.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:53:51.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:51.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get fsid 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:51.173 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:51.174 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:53:51.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:51.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get mon_host 2024-09-15T02:53:51.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-09-15T02:53:57.421 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b) after stopping [dne] 2024-09-15T02:53:57.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:53:57.610 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-09-15T02:53:57.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:53:57.828 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-09-15T02:53:57.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-09-15T02:53:58.012 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-09-15T02:53:58.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:53:58.235 INFO:tasks.workunit.client.0.smithi137.stderr:adding another mon that is not yet online will not break quorum 2024-09-15T02:53:58.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:53:58.495 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.a 2024-09-15T02:53:58.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:53:58.705 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.b 2024-09-15T02:53:58.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-09-15T02:53:58.993 INFO:tasks.workunit.client.0.smithi137.stderr:mon.dne does not exist 2024-09-15T02:53:59.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:59.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:59.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:59.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:59.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:59.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:59.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:59.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:59.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:53:59.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:53:59.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:53:59.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:53:59.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:53:59.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:53:59.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:59.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:53:59.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:53:59.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:59.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:53:59.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:53:59.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:53:59.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:53:59.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:59.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:59.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:53:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:53:59.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:53:59.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:53:59.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:53:59.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:53:59.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:53:59.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:53:59.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:53:59.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:53:59.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:53:59.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:59.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:53:59.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:53:59.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:59.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:53:59.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:53:59.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53903 2024-09-15T02:53:59.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-15T02:53:59.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-15T02:53:59.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:53:59.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:53:59.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:53:59.491 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:53:59.492 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:59.492 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:59.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:53:59.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-15T02:53:59.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:53:59.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:53:59.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:59.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:53:59.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:53:59.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:53:59.711 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:59.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get fsid 2024-09-15T02:53:59.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:53:59.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:53:59.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:53:59.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:53:59.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:53:59.785 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:53:59.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get mon_host 2024-09-15T02:53:59.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-15T02:53:59.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:53:59.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:53:59.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T02:53:59.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:53:59.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-15T02:53:59.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-15T02:54:00.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:54:00.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:00.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:00.195 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:00.195 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:00.195 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:00.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:54:00.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-15T02:54:00.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:00.440 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:00.441 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:00.441 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:54:00.441 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:00.441 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get fsid 2024-09-15T02:54:00.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T02:54:00.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:00.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:54:00.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:00.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get mon_host 2024-09-15T02:54:00.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-15T02:54:00.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:54:00.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:54:00.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-15T02:54:00.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:54:00.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-15T02:54:00.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:54:00.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-15T02:54:01.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:54:01.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-15T02:54:01.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:01.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-15T02:54:01.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:54:01.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:54:01.203 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.c.asok 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:01.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.c.asok config get fsid 2024-09-15T02:54:01.276 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-15T02:54:01.276 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.c.asok 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:01.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.c.asok config get mon_host 2024-09-15T02:54:01.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-09-15T02:54:01.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-15T02:54:01.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-15T02:54:01.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-15T02:54:01.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-15T02:54:01.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-15T02:54:01.350 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-15T02:54:01.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726368901 2024-09-15T02:54:01.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:01.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368841 -lt 1726368901 ]] 2024-09-15T02:54:01.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:01.352 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:04.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:04.536 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:04.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:54:04.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:04.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-15T02:54:04.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-15T02:54:04.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-15T02:54:04.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-15T02:54:04.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-15T02:54:04.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-09-15T02:54:04.748 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,c) after stopping [dne] 2024-09-15T02:54:04.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:54:05.027 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c) after stopping [a] 2024-09-15T02:54:05.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:54:05.361 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-15T02:54:05.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-15T02:54:05.660 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b) after stopping [c] 2024-09-15T02:54:05.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-09-15T02:54:05.889 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-09-15T02:54:05.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-09-15T02:54:09.030 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-09-15T02:54:09.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-09-15T02:54:09.242 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-09-15T02:54:09.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:54:09.516 INFO:tasks.workunit.client.0.smithi137.stderr:adding another mon that is not yet online will not break quorum 2024-09-15T02:54:09.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:54:09.735 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.a 2024-09-15T02:54:09.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:54:10.125 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.b 2024-09-15T02:54:10.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-15T02:54:10.408 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.c 2024-09-15T02:54:10.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-09-15T02:54:10.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:54:10.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:54:10.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:54:10.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:54:10.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-15T02:54:10.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-15T02:54:10.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726368910 2024-09-15T02:54:10.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:10.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368850 -lt 1726368910 ]] 2024-09-15T02:54:10.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:10.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:13.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:13.710 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:13.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:13.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:13.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:13.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:13.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:13.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368853 -lt 1726368910 ]] 2024-09-15T02:54:13.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:13.715 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:13.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:13.893 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:13.894 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:13.894 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:13.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:13.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:13.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:13.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368853 -lt 1726368910 ]] 2024-09-15T02:54:13.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:13.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:14.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:14.075 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.075 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:14.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:14.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:14.078 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:14.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368854 -lt 1726368910 ]] 2024-09-15T02:54:14.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:14.079 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:14.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:14.257 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:14.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:14.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:14.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:14.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368854 -lt 1726368910 ]] 2024-09-15T02:54:14.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:14.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:14.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:14.437 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:14.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:14.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:14.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:14.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368854 -lt 1726368910 ]] 2024-09-15T02:54:14.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:14.442 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:14.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:14.619 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:14.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:14.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:14.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:14.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368854 -lt 1726368910 ]] 2024-09-15T02:54:14.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:14.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.801 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.801 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:14.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:14.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:14.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:14.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368854 -lt 1726368910 ]] 2024-09-15T02:54:14.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:14.805 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:14.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:14.982 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:14.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:14.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:14.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:14.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:14.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368854 -lt 1726368910 ]] 2024-09-15T02:54:14.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:14.987 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:15.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:15.165 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:15.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:15.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:15.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:15.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368855 -lt 1726368910 ]] 2024-09-15T02:54:15.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:15.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:15.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:15.346 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:15.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:15.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:15.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:15.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368855 -lt 1726368910 ]] 2024-09-15T02:54:15.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:15.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:15.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:15.528 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:15.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:15.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:15.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:15.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368855 -lt 1726368910 ]] 2024-09-15T02:54:15.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:15.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:15.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:15.710 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:15.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:15.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:15.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:15.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368855 -lt 1726368910 ]] 2024-09-15T02:54:15.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:15.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:15.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:15.891 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.892 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:15.892 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:15.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:15.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:15.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:15.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368855 -lt 1726368910 ]] 2024-09-15T02:54:15.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:15.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:16.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:16.073 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:16.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:16.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:16.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:16.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:16.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368856 -lt 1726368910 ]] 2024-09-15T02:54:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:16.078 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:16.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:16.255 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:16.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:16.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:16.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:16.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:16.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368856 -lt 1726368910 ]] 2024-09-15T02:54:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:16.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:16.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:16.437 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:16.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:16.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:16.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:16.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:16.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:16.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368856 -lt 1726368910 ]] 2024-09-15T02:54:16.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:16.442 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:19.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:19.629 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:19.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:19.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:19.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:19.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:19.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:19.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368859 -lt 1726368910 ]] 2024-09-15T02:54:19.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:19.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:19.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:19.812 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:19.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:19.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:19.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:19.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:19.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:19.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368859 -lt 1726368910 ]] 2024-09-15T02:54:19.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:19.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:19.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:19.991 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:19.992 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:19.992 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:19.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:19.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:19.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:19.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368859 -lt 1726368910 ]] 2024-09-15T02:54:19.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:19.996 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:20.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:20.173 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:20.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:20.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:20.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:20.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:20.175 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:20.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368860 -lt 1726368910 ]] 2024-09-15T02:54:20.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:20.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:20.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:20.353 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:20.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:20.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:20.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:20.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:20.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:20.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368860 -lt 1726368910 ]] 2024-09-15T02:54:20.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-15T02:54:20.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:32.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:32.543 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:32.543 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:53:59.149142Z","created":"2024-09-15T02:53:59.149142Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:32.544 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-15T02:54:32.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-15T02:54:32.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-15T02:54:32.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-15T02:54:32.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-15T02:54:32.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-15T02:54:32.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:54:32.773 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-09-15T02:54:32.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:54:35.949 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-15T02:54:35.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-15T02:54:36.147 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-09-15T02:54:36.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:54:39.285 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-15T02:54:39.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:54:39.516 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-15T02:54:39.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:54:39.775 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.b 2024-09-15T02:54:39.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-15T02:54:42.919 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-15T02:54:42.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-15T02:54:42.922 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:54:42.922 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:54:42.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:54:42.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:54:42.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:54:43.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:54:43.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-15T02:54:43.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:54:43.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:54:43.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:54:43.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:54:43.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:54:43.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:54:43.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:54:43.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:54:43.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:54:43.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:54:43.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:54:43.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:54:43.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:54:43.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:54:43.036 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:54:43.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:54:43.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:54:43.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:54:43.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:54:43.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:54:43.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:54:43.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:54:43.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:54:43.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:43.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:43.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:54:43.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:54:43.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:54:43.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-15T02:54:43.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-15T02:54:43.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-15T02:54:43.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-15T02:54:43.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:54:43.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:54:43.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:54:43.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:54:43.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:54:43.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:54:43.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:54:43.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:54:43.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:54:43.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:54:43.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:54:43.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:54:43.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:54:43.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:54:43.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:54:43.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:54:43.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:43.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:43.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:54:43.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:54:43.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:54:43.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-15T02:54:43.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:54:43.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:43.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:43.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53903 2024-09-15T02:54:43.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:54:43.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:54:43.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:54:43.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-15T02:54:43.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:54:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-15T02:54:43.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:54:43.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:54:43.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:43.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:54:43.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:54:43.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:43.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get fsid 2024-09-15T02:54:43.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:54:43.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:43.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:54:43.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:54:43.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:54:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:43.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get mon_host 2024-09-15T02:54:43.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-15T02:54:43.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:54:43.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:54:43.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T02:54:43.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:54:43.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-15T02:54:43.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-15T02:54:44.137 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:54:44.137 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:44.137 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:44.137 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:44.137 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:44.138 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:44.138 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:54:44.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-15T02:54:44.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:44.416 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:44.417 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:54:44.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:44.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get fsid 2024-09-15T02:54:44.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:54:44.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:44.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get mon_host 2024-09-15T02:54:44.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-15T02:54:44.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:54:44.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:54:44.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-15T02:54:44.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:54:44.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-15T02:54:44.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:54:44.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-15T02:54:45.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:45.162 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.c.asok 2024-09-15T02:54:45.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:45.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.c.asok config get fsid 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-15T02:54:45.235 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-15T02:54:45.236 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:45.236 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:45.236 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:45.236 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.c.asok 2024-09-15T02:54:45.236 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:45.236 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.c.asok config get mon_host 2024-09-15T02:54:45.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-15T02:54:45.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:54:45.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:54:45.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-15T02:54:45.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:54:45.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-15T02:54:45.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-15T02:54:45.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:54:45.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:45.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:45.662 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:45.662 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:45.662 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:45.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:54:45.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-15T02:54:45.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:54:45.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-15T02:54:45.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:45.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-15T02:54:45.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:54:45.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:54:45.941 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-15T02:54:45.941 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-15T02:54:45.941 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-15T02:54:45.942 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:45.942 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:45.942 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:45.942 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.d.asok 2024-09-15T02:54:45.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:45.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.d.asok config get fsid 2024-09-15T02:54:46.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-15T02:54:46.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.d.asok 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:54:46.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.d.asok config get mon_host 2024-09-15T02:54:46.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-09-15T02:54:46.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-15T02:54:46.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-15T02:54:46.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-15T02:54:46.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-15T02:54:46.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-15T02:54:46.090 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-15T02:54:46.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726368946 2024-09-15T02:54:46.091 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:46.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368886 -lt 1726368946 ]] 2024-09-15T02:54:46.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:54:46.091 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:55.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:55.280 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:55.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:54:55.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:55.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-15T02:54:55.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-15T02:54:55.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-15T02:54:55.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-15T02:54:55.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-15T02:54:55.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-09-15T02:54:55.515 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,c,d) after stopping [dne] 2024-09-15T02:54:55.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:54:55.758 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-15T02:54:55.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:54:56.025 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-09-15T02:54:56.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-15T02:54:56.322 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-09-15T02:54:56.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-15T02:54:56.776 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-09-15T02:54:56.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-09-15T02:54:57.028 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-09-15T02:54:57.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-09-15T02:54:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-09-15T02:54:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:54:57.520 INFO:tasks.workunit.client.0.smithi137.stderr:adding another mon that is not yet online will not break quorum 2024-09-15T02:54:57.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:54:57.765 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.a 2024-09-15T02:54:57.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:54:58.177 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.b 2024-09-15T02:54:58.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-15T02:54:58.573 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.c 2024-09-15T02:54:58.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-15T02:54:58.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:54:58.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:54:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:54:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:54:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:54:58.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:54:58.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-09-15T02:54:58.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-15T02:54:58.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-15T02:54:58.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-15T02:54:58.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-15T02:54:58.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-15T02:54:58.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-15T02:54:58.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726368958 2024-09-15T02:54:58.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:58.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368898 -lt 1726368958 ]] 2024-09-15T02:54:58.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:58.690 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:58.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:58.868 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:58.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:58.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:54:58.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:58.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:58.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:58.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368898 -lt 1726368958 ]] 2024-09-15T02:54:58.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:58.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:59.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:59.050 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:54:59.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:59.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:59.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:59.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368899 -lt 1726368958 ]] 2024-09-15T02:54:59.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:59.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:59.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:59.232 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.232 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:54:59.232 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:59.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:59.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:59.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368899 -lt 1726368958 ]] 2024-09-15T02:54:59.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:59.236 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:59.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:59.414 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.414 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.414 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:54:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:59.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:59.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368899 -lt 1726368958 ]] 2024-09-15T02:54:59.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:59.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:59.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:59.607 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:54:59.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:59.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:59.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368899 -lt 1726368958 ]] 2024-09-15T02:54:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:59.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:54:59.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:54:59.788 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.789 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:54:59.789 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:54:59.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:54:59.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:54:59.792 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:54:59.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368899 -lt 1726368958 ]] 2024-09-15T02:54:59.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:54:59.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:02.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:02.977 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:02.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:02.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:02.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:02.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:02.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:02.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368902 -lt 1726368958 ]] 2024-09-15T02:55:02.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:02.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:03.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:03.160 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:03.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:03.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:03.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:03.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368903 -lt 1726368958 ]] 2024-09-15T02:55:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:03.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:03.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:03.342 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:03.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:03.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:03.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:03.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368903 -lt 1726368958 ]] 2024-09-15T02:55:03.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:03.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:03.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:03.523 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:03.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:03.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:03.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:03.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368903 -lt 1726368958 ]] 2024-09-15T02:55:03.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:03.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:03.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:03.706 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:03.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:03.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:03.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:03.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368903 -lt 1726368958 ]] 2024-09-15T02:55:03.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:03.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:03.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:03.886 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.887 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:03.887 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:03.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:03.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:03.889 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:03.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368903 -lt 1726368958 ]] 2024-09-15T02:55:03.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:03.890 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:04.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:04.067 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:04.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:04.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:04.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:04.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:04.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:04.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368904 -lt 1726368958 ]] 2024-09-15T02:55:04.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:04.073 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:07.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:07.254 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:07.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:07.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:07.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:07.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368907 -lt 1726368958 ]] 2024-09-15T02:55:07.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:07.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:07.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:07.452 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.453 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:07.453 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:07.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:07.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:07.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368907 -lt 1726368958 ]] 2024-09-15T02:55:07.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:07.457 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:07.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:07.639 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.640 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.640 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:07.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:07.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:07.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:07.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368907 -lt 1726368958 ]] 2024-09-15T02:55:07.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:07.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:07.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:07.820 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:07.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:07.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:07.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:07.823 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:07.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368907 -lt 1726368958 ]] 2024-09-15T02:55:07.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:07.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:14.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:14.075 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:14.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:14.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:54:43.075796Z","created":"2024-09-15T02:54:43.075796Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:14.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-15T02:55:14.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-15T02:55:14.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-15T02:55:14.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-15T02:55:14.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-15T02:55:14.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:55:14.306 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-15T02:55:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:55:14.519 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-15T02:55:14.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-15T02:55:17.659 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-15T02:55:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-15T02:55:17.897 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-15T02:55:17.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:55:18.164 INFO:tasks.workunit.client.0.smithi137.stderr:adding another mon that is not yet online will not break quorum 2024-09-15T02:55:18.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:55:18.501 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.a 2024-09-15T02:55:18.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:55:18.752 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.b 2024-09-15T02:55:18.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-15T02:55:18.970 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.c 2024-09-15T02:55:18.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-09-15T02:55:19.332 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.d 2024-09-15T02:55:19.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-15T02:55:19.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:55:19.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:55:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:55:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:55:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:55:19.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:55:19.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-15T02:55:19.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:55:19.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:55:19.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:55:19.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:55:19.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:55:19.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:55:19.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:55:19.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:55:19.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:55:19.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:55:19.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:55:19.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:55:19.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:55:19.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:55:19.456 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:55:19.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:55:19.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:55:19.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:55:19.457 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:55:19.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:55:19.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:55:19.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:55:19.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:55:19.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:19.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:19.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:55:19.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:55:19.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:55:19.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:55:19.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:55:19.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:55:19.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:55:19.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:55:19.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:55:19.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:55:19.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:55:19.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:55:19.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:55:19.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:55:19.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:55:19.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:55:19.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:55:19.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:55:19.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:55:19.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:19.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:19.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:55:19.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:55:19.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:55:19.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-15T02:55:19.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:55:19.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:19.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:19.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53903 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=6a2f3c11-0338-4cd4-b066-ed53fc38051e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:55:19.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:55:19.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:55:19.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:55:19.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-15T02:55:19.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-15T02:55:19.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:55:19.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:55:19.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:55:19.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:55:19.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:19.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:19.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:55:19.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-15T02:55:20.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:55:20.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:55:20.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:20.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:55:20.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:55:20.075 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:55:20.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:55:20.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:55:20.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:55:20.076 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:20.076 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:20.076 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:20.076 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:55:20.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:20.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get fsid 2024-09-15T02:55:20.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:55:20.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:20.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:55:20.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:55:20.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:55:20.151 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.a.asok 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:20.152 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.a.asok config get mon_host 2024-09-15T02:55:20.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-15T02:55:20.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:55:20.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:55:20.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T02:55:20.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:55:20.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-15T02:55:20.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-15T02:55:20.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:55:20.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:55:20.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:55:20.584 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:55:20.584 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:20.584 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:20.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:55:20.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-15T02:55:20.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:55:20.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T02:55:20.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:20.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:55:20.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:55:20.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:55:20.837 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:55:20.837 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:55:20.837 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:55:20.838 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:20.838 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:20.838 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:20.838 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:55:20.838 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:20.838 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get fsid 2024-09-15T02:55:20.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T02:55:20.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:20.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T02:55:20.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.b.asok 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:20.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.b.asok config get mon_host 2024-09-15T02:55:20.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-15T02:55:20.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:55:20.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:55:20.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-15T02:55:20.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:55:20.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-15T02:55:20.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-15T02:55:21.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:55:21.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:55:21.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:55:21.338 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:55:21.338 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:21.338 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:21.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:55:21.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-15T02:55:21.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-15T02:55:21.592 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:21.593 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:21.593 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:21.593 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.c.asok 2024-09-15T02:55:21.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:21.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.c.asok config get fsid 2024-09-15T02:55:21.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-15T02:55:21.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:21.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-15T02:55:21.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:55:21.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.c.asok 2024-09-15T02:55:21.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:21.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.c.asok config get mon_host 2024-09-15T02:55:21.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-15T02:55:21.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:55:21.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:55:21.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-15T02:55:21.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:55:21.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-15T02:55:21.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-15T02:55:22.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:55:22.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:55:22.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:55:22.093 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:55:22.093 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:22.093 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:22.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:55:22.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-15T02:55:22.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:55:22.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-15T02:55:22.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:22.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-15T02:55:22.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:55:22.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:55:22.371 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-15T02:55:22.371 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-15T02:55:22.371 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-15T02:55:22.371 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:22.372 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:22.372 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:22.372 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.d.asok 2024-09-15T02:55:22.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:22.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.d.asok config get fsid 2024-09-15T02:55:22.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-15T02:55:22.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:22.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-15T02:55:22.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:55:22.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.d.asok 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:22.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.d.asok config get mon_host 2024-09-15T02:55:22.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-09-15T02:55:22.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-15T02:55:22.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:55:22.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-09-15T02:55:22.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:55:22.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-09-15T02:55:22.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-09-15T02:55:22.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:55:22.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:55:22.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:55:22.879 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:55:22.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:22.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:22.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53903/$cluster-$name.asok' 2024-09-15T02:55:22.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53903/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-09-15T02:55:23.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:55:23.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:23.234 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:23.235 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.e.asok 2024-09-15T02:55:23.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:23.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.e.asok config get fsid 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:55:23.308 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:23.309 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:55:23.309 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53903/ceph-mon.e.asok 2024-09-15T02:55:23.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:55:23.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53903/ceph-mon.e.asok config get mon_host 2024-09-15T02:55:23.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-09-15T02:55:23.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-15T02:55:23.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-09-15T02:55:23.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-15T02:55:23.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-09-15T02:55:23.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-15T02:55:23.381 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-15T02:55:23.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726368983 2024-09-15T02:55:23.381 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:23.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368923 -lt 1726368983 ]] 2024-09-15T02:55:23.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-15T02:55:23.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:29.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:29.571 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:29.572 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-09-15T02:55:29.572 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-15T02:55:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-15T02:55:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-15T02:55:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-15T02:55:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-15T02:55:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-09-15T02:55:29.841 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,c,d,e) after stopping [dne] 2024-09-15T02:55:29.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:55:30.250 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-15T02:55:30.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:55:30.536 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-09-15T02:55:30.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-15T02:55:30.797 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-09-15T02:55:30.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-15T02:55:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-09-15T02:55:31.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-15T02:55:31.350 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-09-15T02:55:31.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-09-15T02:55:31.637 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-09-15T02:55:31.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-09-15T02:55:31.889 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-09-15T02:55:31.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-09-15T02:55:32.169 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-09-15T02:55:32.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:55:32.340 INFO:tasks.workunit.client.0.smithi137.stderr:adding another mon that is not yet online will not break quorum 2024-09-15T02:55:32.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:55:32.539 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.a 2024-09-15T02:55:32.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:55:32.948 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.b 2024-09-15T02:55:32.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-15T02:55:33.224 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.c 2024-09-15T02:55:33.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-15T02:55:33.567 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.d 2024-09-15T02:55:33.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-15T02:55:33.844 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.e 2024-09-15T02:55:33.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-15T02:55:33.853 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:55:33.854 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:55:33.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:55:33.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:55:33.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:55:33.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-15T02:55:33.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726368993 2024-09-15T02:55:33.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:33.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368933 -lt 1726368993 ]] 2024-09-15T02:55:33.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:33.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:34.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:34.138 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:34.138 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:34.138 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:34.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:34.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:34.141 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:34.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368934 -lt 1726368993 ]] 2024-09-15T02:55:34.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:34.142 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:34.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:34.319 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:34.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:34.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:34.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:34.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:34.322 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:34.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368934 -lt 1726368993 ]] 2024-09-15T02:55:34.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:34.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:37.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:37.510 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:37.510 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:37.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:37.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:37.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:37.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:37.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368937 -lt 1726368993 ]] 2024-09-15T02:55:37.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:37.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:37.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:37.691 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:37.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:37.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:37.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:37.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:37.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:37.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368937 -lt 1726368993 ]] 2024-09-15T02:55:37.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:37.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:40.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:40.880 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:40.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:40.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:40.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:40.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:40.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:40.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368940 -lt 1726368993 ]] 2024-09-15T02:55:40.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:40.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:41.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:41.129 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:41.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:41.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:41.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:41.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:41.132 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:41.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368941 -lt 1726368993 ]] 2024-09-15T02:55:41.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:41.133 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:41.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:41.310 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:41.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:41.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:41.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:41.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:41.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:41.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368941 -lt 1726368993 ]] 2024-09-15T02:55:41.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-15T02:55:41.314 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:50.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:50.500 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:50.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-15T02:55:50.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:50.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-15T02:55:50.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-15T02:55:50.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-15T02:55:50.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-15T02:55:50.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-15T02:55:50.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:55:50.755 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-15T02:55:50.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:55:51.035 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-09-15T02:55:51.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-15T02:55:51.335 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-09-15T02:55:51.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-15T02:55:51.596 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-09-15T02:55:51.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-15T02:55:51.846 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-15T02:55:51.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:55:52.099 INFO:tasks.workunit.client.0.smithi137.stderr:adding another mon that is not yet online will not break quorum 2024-09-15T02:55:52.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:55:52.376 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.a 2024-09-15T02:55:52.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:55:52.645 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.b 2024-09-15T02:55:52.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-15T02:55:52.905 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.c 2024-09-15T02:55:52.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-15T02:55:53.140 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.d 2024-09-15T02:55:53.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-15T02:55:53.415 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.e 2024-09-15T02:55:53.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-09-15T02:55:53.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:55:53.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:55:53.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:55:53.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:55:53.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-15T02:55:53.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-15T02:55:53.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726369013 2024-09-15T02:55:53.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:53.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368953 -lt 1726369013 ]] 2024-09-15T02:55:53.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:53.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:53.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:53.712 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:53.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:53.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:53.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:53.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:53.715 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:53.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368953 -lt 1726369013 ]] 2024-09-15T02:55:53.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:53.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:56.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:56.903 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:56.903 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:56.904 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:56.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:56.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:56.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:56.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368956 -lt 1726369013 ]] 2024-09-15T02:55:56.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:56.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:55:57.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:55:57.086 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:57.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:55:57.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:55:57.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:55:57.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:55:57.089 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:55:57.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368957 -lt 1726369013 ]] 2024-09-15T02:55:57.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:55:57.090 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:56:00.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:56:00.276 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.276 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.276 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:56:00.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:56:00.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:56:00.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:56:00.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368960 -lt 1726369013 ]] 2024-09-15T02:56:00.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:56:00.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:56:00.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:56:00.461 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:56:00.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:56:00.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:56:00.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:56:00.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368960 -lt 1726369013 ]] 2024-09-15T02:56:00.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:56:00.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:56:00.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:56:00.643 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:56:00.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:56:00.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:56:00.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:56:00.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368960 -lt 1726369013 ]] 2024-09-15T02:56:00.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:56:00.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:56:00.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:56:00.825 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:56:00.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:00.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-15T02:56:00.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-15T02:56:00.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-15T02:56:00.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726368960 -lt 1726369013 ]] 2024-09-15T02:56:00.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-15T02:56:00.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-15T02:56:10.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-15T02:56:10.016 INFO:tasks.workunit.client.0.smithi137.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:10.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"6a2f3c11-0338-4cd4-b066-ed53fc38051e","modified":"2024-09-15T02:55:19.494164Z","created":"2024-09-15T02:55:19.494164Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-15T02:56:10.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-15T02:56:10.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-15T02:56:10.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-15T02:56:10.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-15T02:56:10.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-15T02:56:10.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-15T02:56:10.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-15T02:56:10.241 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-15T02:56:10.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-15T02:56:10.561 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-15T02:56:10.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-15T02:56:10.746 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-15T02:56:10.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-15T02:56:10.905 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-15T02:56:10.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-15T02:56:14.084 INFO:tasks.workunit.client.0.smithi137.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-15T02:56:14.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-15T02:56:14.317 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-15T02:56:14.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-15T02:56:14.576 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.a 2024-09-15T02:56:14.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-15T02:56:14.781 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: removing mon.b would break quorum 2024-09-15T02:56:14.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-15T02:56:14.987 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-15T02:56:14.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-15T02:56:15.239 INFO:tasks.workunit.client.0.smithi137.stderr:Error EBUSY: removing mon.d would break quorum 2024-09-15T02:56:15.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-15T02:56:15.474 INFO:tasks.workunit.client.0.smithi137.stderr:safe to remove mon.e 2024-09-15T02:56:15.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-15T02:56:15.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:56:15.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:56:15.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:56:15.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:56:15.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:56:15.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:56:15.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:56:15.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:56:15.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:56:15.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:56:15.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:56:15.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:56:15.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:56:15.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:15.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:56:15.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:56:15.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:15.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:56:15.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:56:15.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:56:15.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:56:15.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:15.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:56:15.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:56:15.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:56:15.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:56:15.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/ok-to-stop 0 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:56:15.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:56:15.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:56:15.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:56:15.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:56:15.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:56:15.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:56:15.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:56:15.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:56:15.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:15.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:56:15.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:56:15.617 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:15.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:56:15.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T02:56:15.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-15T02:56:15.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:56:15.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:15.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53903 2024-09-15T02:56:15.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53903 2024-09-15T02:56:15.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:56:15.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:56:15.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T02:56:15.622 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:56:15.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:56:15.687 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-09-15T02:56:15.687 DEBUG:teuthology.orchestra.run.smithi137:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-09-15T02:56:15.748 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:56:15.755 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:56:15.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:56:15.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:56:15.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T02:56:15.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T02:56:15.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T02:56:15.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/rados-striper 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none ' 2024-09-15T02:56:15.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:56:15.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:56:15.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:56:15.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:56:15.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:56:15.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:56:15.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:56:15.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:56:15.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:56:15.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:15.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:56:15.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:56:15.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:15.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:56:15.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:56:15.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-15T02:56:15.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:56:15.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:15.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:56:15.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71136 2024-09-15T02:56:15.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:56:15.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:56:15.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-09-15T02:56:15.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:56:15.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:15.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:56:15.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71136 2024-09-15T02:56:15.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:56:15.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:56:15.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:56:15.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:56:15.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-09-15T02:56:15.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-09-15T02:56:15.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-09-15T02:56:15.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:56:15.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:56:15.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:56:15.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-09-15T02:56:15.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-09-15T02:56:16.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:56:16.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:56:16.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:56:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:56:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:56:16.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71136/$cluster-$name.asok' 2024-09-15T02:56:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:56:16.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:56:16.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:56:16.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:56:16.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:56:16.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:56:16.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71136/ceph-mon.a.asok 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:56:16.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71136/ceph-mon.a.asok config get fsid 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:16.418 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:16.419 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:56:16.419 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71136/ceph-mon.a.asok 2024-09-15T02:56:16.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:56:16.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71136/ceph-mon.a.asok config get mon_host 2024-09-15T02:56:16.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-09-15T02:56:16.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:56:16.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71136/$cluster-$name.asok' 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' 2024-09-15T02:56:16.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:56:16.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=67e4092a-c2b2-482a-8631-4bbedbf24788 2024-09-15T02:56:16.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 67e4092a-c2b2-482a-8631-4bbedbf24788' 2024-09-15T02:56:16.498 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 67e4092a-c2b2-482a-8631-4bbedbf24788 2024-09-15T02:56:16.498 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:56:16.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQTOZm2goqHhAAMh+bR+uLhuDDrItHovjarA== 2024-09-15T02:56:16.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQTOZm2goqHhAAMh+bR+uLhuDDrItHovjarA=="}' 2024-09-15T02:56:16.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 67e4092a-c2b2-482a-8631-4bbedbf24788 -i td/rados-striper/0/new.json 2024-09-15T02:56:17.371 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:56:17.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-09-15T02:56:17.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQTOZm2goqHhAAMh+bR+uLhuDDrItHovjarA== --osd-uuid 67e4092a-c2b2-482a-8631-4bbedbf24788 2024-09-15T02:56:17.404 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:17.402+0000 7f067357a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:17.425 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:17.424+0000 7f067357a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:17.426 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:17.425+0000 7f067357a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:17.427 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:17.425+0000 7f067357a440 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-15T02:56:17.427 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:17.425+0000 7f067357a440 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-15T02:56:17.427 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:17.425+0000 7f067357a440 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-15T02:56:17.466 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:17.465+0000 7f067357a440 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-09-15T02:56:22.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-09-15T02:56:22.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:56:22.801 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:56:22.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:56:22.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:56:22.995 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:56:22.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:56:22.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:56:22.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:56:22.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:56:22.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:56:23.015 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:23.014+0000 7fe233c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:23.035 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:23.033+0000 7fe233c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:23.036 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:23.034+0000 7fe233c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:56:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:56:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:56:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:56:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:56:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:56:23.207 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:56:23.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:23.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:23.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:23.564 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:23.563+0000 7fe233c0c440 -1 Falling back to public interface 2024-09-15T02:56:24.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:24.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:24.409 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:56:24.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:56:24.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:24.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:24.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:25.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:25.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:25.593 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:56:25.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:56:25.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:25.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:25.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:26.127 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:26.126+0000 7fe233c0c440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:56:26.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:26.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:26.777 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:56:26.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:56:26.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:26.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:27.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:28.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:28.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:28.083 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:56:28.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:56:28.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:28.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:28.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:29.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:29.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:29.267 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:56:29.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:56:29.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:29.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:29.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:30.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:30.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:30.452 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:56:30.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:56:30.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:30.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:30.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:31.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:31.635 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:56:31.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:31.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:56:31.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:31.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:31.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:32.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:32.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:32.822 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:56:32.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:56:32.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:32.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:33.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:34.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:34.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:34.007 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:56:34.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:56:34.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:34.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:34.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:35.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:35.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:35.195 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:56:35.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:56:35.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:35.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:35.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:36.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:36.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:36.382 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:56:36.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:56:36.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:36.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:36.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:37.567 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:56:37.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:56:37.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:37.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:37.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:38.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:38.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:38.752 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:56:38.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:56:38.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:38.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:38.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:39.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:39.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:39.937 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:56:39.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:56:39.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:39.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:40.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:41.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:41.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:41.116 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:56:41.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:56:41.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:41.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:41.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:42.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:42.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:42.301 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:56:42.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:56:42.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:42.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:42.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:43.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:43.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:43.485 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:56:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:56:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:43.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:43.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:44.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:44.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:44.669 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T02:56:44.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:56:44.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:44.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:44.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:45.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:45.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:45.853 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T02:56:45.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T02:56:45.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:45.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:46.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:47.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:47.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:47.031 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T02:56:47.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T02:56:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:47.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:47.210 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/72078,v1:127.0.0.1:6801/72078] [v2:127.0.0.1:6802/72078,v1:127.0.0.1:6803/72078] exists,up 67e4092a-c2b2-482a-8631-4bbedbf24788 2024-09-15T02:56:47.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:56:47.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:56:47.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:56:47.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71136/$cluster-$name.asok' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:56:47.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:56:47.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-09-15T02:56:47.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:56:47.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e482d49-8e94-4949-924d-db35ce53f4e6 2024-09-15T02:56:47.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0e482d49-8e94-4949-924d-db35ce53f4e6' 2024-09-15T02:56:47.217 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 0e482d49-8e94-4949-924d-db35ce53f4e6 2024-09-15T02:56:47.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:56:47.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvTOZmyrxsDRAANuYBagrHCCKS6lybqr7PIA== 2024-09-15T02:56:47.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvTOZmyrxsDRAANuYBagrHCCKS6lybqr7PIA=="}' 2024-09-15T02:56:47.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e482d49-8e94-4949-924d-db35ce53f4e6 -i td/rados-striper/1/new.json 2024-09-15T02:56:47.682 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:56:47.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-09-15T02:56:47.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvTOZmyrxsDRAANuYBagrHCCKS6lybqr7PIA== --osd-uuid 0e482d49-8e94-4949-924d-db35ce53f4e6 2024-09-15T02:56:47.715 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:47.714+0000 7f6100d2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:47.716 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:47.716+0000 7f6100d2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:47.717 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:47.717+0000 7f6100d2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:47.718 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:47.717+0000 7f6100d2d440 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-15T02:56:47.718 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:47.717+0000 7f6100d2d440 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-15T02:56:47.718 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:47.717+0000 7f6100d2d440 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-15T02:56:47.795 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:47.795+0000 7f6100d2d440 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-09-15T02:56:53.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-09-15T02:56:53.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:56:53.373 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T02:56:53.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:56:53.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:56:53.564 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T02:56:53.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:56:53.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:56:53.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:56:53.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:56:53.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:56:53.585 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:53.585+0000 7f24a4d1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:53.597 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:53.596+0000 7f24a4d1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:53.598 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:53.597+0000 7f24a4d1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:53.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:56:53.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:56:53.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:56:53.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:56:53.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:56:53.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:53.769 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:56:53.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:56:53.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:53.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:56:53.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:54.126 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:54.126+0000 7f24a4d1e440 -1 Falling back to public interface 2024-09-15T02:56:54.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:54.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:54.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:56:54.978 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:56:54.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:54.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:56:55.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:56.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:56.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:56.155 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:56:56.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:56:56.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:56.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:56:56.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:56.673 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:56:56.672+0000 7f24a4d1e440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:56:57.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:57.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:57.337 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:56:57.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:56:57.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:57.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:56:57.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:58.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:58.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:58.687 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:56:58.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:56:58.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:58.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:56:58.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:59.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:59.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:59.871 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:56:59.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:56:59.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:59.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:00.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:01.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:01.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:01.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:57:01.056 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:57:01.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:01.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:01.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:02.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:02.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:02.240 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:57:02.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:57:02.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:02.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:02.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:03.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:03.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:03.424 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:57:03.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:57:03.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:03.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:03.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:04.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:04.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:04.608 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:57:04.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:57:04.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:04.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:04.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:05.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:05.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:05.796 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:57:05.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:57:05.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:05.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:05.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:06.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:06.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:06.978 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:57:06.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:57:06.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:06.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:07.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:08.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:08.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:08.161 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:57:08.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:57:08.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:08.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:08.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:09.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:09.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:09.347 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:57:09.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:57:09.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:09.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:09.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:10.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:10.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:10.530 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:57:10.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:57:10.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:10.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:10.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:11.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:11.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:11.716 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:57:11.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:57:11.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:11.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:11.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:12.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:12.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:12.904 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:57:12.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:57:12.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:12.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:13.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:14.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:14.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:14.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:57:14.085 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:57:14.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:14.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:14.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:15.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:15.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:15.273 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T02:57:15.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:57:15.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:15.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:15.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:16.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:16.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:16.452 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T02:57:16.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T02:57:16.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:16.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:16.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:17.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:17.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:17.637 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T02:57:17.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T02:57:17.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:17.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:17.824 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/73908,v1:127.0.0.1:6809/73908] [v2:127.0.0.1:6810/73908,v1:127.0.0.1:6811/73908] exists,up 0e482d49-8e94-4949-924d-db35ce53f4e6 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:57:17.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:57:17.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71136/$cluster-$name.asok' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-15T02:57:17.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2082732c-f97b-442d-b569-11675e045c1e 2024-09-15T02:57:17.831 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 2082732c-f97b-442d-b569-11675e045c1e 2024-09-15T02:57:17.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2082732c-f97b-442d-b569-11675e045c1e' 2024-09-15T02:57:17.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:57:17.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANTeZmoFsQMhAAMaP3FS/yj68kx1fNgz2ovw== 2024-09-15T02:57:17.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANTeZmoFsQMhAAMaP3FS/yj68kx1fNgz2ovw=="}' 2024-09-15T02:57:17.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2082732c-f97b-442d-b569-11675e045c1e -i td/rados-striper/2/new.json 2024-09-15T02:57:18.231 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:57:18.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-09-15T02:57:18.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANTeZmoFsQMhAAMaP3FS/yj68kx1fNgz2ovw== --osd-uuid 2082732c-f97b-442d-b569-11675e045c1e 2024-09-15T02:57:18.264 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:18.262+0000 7fe1af927440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:18.266 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:18.264+0000 7fe1af927440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:18.266 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:18.265+0000 7fe1af927440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:18.267 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:18.266+0000 7fe1af927440 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-15T02:57:18.267 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:18.266+0000 7fe1af927440 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-15T02:57:18.267 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:18.266+0000 7fe1af927440 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-15T02:57:18.341 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:18.339+0000 7fe1af927440 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-09-15T02:57:23.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-09-15T02:57:23.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:57:23.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:57:23.931 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T02:57:23.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:57:24.120 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T02:57:24.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:57:24.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=252ead3a-67bb-469e-b0e7-42b61b778e30 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.71136/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:57:24.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:57:24.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:57:24.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:57:24.141 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:24.140+0000 7fe5ede13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:24.152 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:24.150+0000 7fe5ede13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:24.153 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:24.152+0000 7fe5ede13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:24.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:57:24.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:57:24.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:57:24.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:57:24.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:57:24.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:24.323 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:57:24.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:57:24.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:24.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:24.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:24.684 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:24.683+0000 7fe5ede13440 -1 Falling back to public interface 2024-09-15T02:57:25.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:25.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:25.533 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:57:25.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:57:25.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:25.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:25.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:26.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:26.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:26.717 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:57:26.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:57:26.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:26.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:26.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:27.203 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:27.202+0000 7fe5ede13440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:27.901 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:57:27.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:57:27.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:27.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:28.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:29.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:29.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:29.191 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:57:29.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:57:29.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:29.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:29.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:30.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:30.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:30.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:57:30.375 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:57:30.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:30.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:30.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:31.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:31.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:31.560 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:57:31.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:57:31.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:31.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:31.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:32.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:32.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:32.743 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:57:32.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:57:32.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:32.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:32.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:33.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:33.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:33.929 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:57:33.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:57:33.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:33.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:34.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:35.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:35.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:35.114 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:57:35.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:57:35.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:35.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:35.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:36.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:36.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:36.301 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:57:36.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:57:36.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:36.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:36.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:37.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:37.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:37.486 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:57:37.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:57:37.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:37.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:37.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:38.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:38.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:38.673 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:57:38.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:57:38.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:38.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:38.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:39.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:39.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:39.857 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:57:39.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:57:39.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:39.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:40.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:41.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:41.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:41.041 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:57:41.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:57:41.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:41.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:41.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:42.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:42.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:42.225 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:57:42.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:57:42.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:42.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:42.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:43.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:43.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:43.411 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:57:43.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:57:43.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:43.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:43.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:44.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:44.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:44.595 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:57:44.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:57:44.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:44.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:44.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:45.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:45.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:45.778 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T02:57:45.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:57:45.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:45.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:45.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:46.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:46.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:46.957 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T02:57:46.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T02:57:46.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:46.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:47.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:48.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:48.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:48.139 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T02:57:48.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T02:57:48.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:48.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:48.381 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/75737,v1:127.0.0.1:6817/75737] [v2:127.0.0.1:6818/75737,v1:127.0.0.1:6819/75737] exists,up 2082732c-f97b-442d-b569-11675e045c1e 2024-09-15T02:57:48.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:57:48.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:57:48.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:57:48.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-09-15T02:57:48.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T02:57:48.550 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T02:57:48.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T02:57:48.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:57:48.943 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T02:57:48.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:57:49.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T02:57:52.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-09-15T02:57:52.016 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-09-15T02:57:52.016 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-09-15T02:57:52.016 INFO:tasks.workunit.client.0.smithi137.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 5.8285e-05 s, 21.2 MB/s 2024-09-15T02:57:52.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-09-15T02:57:52.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-09-15T02:57:52.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-09-15T02:57:52.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-15T02:57:52.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-09-15T02:57:52.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-09-15T02:57:52.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-09-15T02:57:52.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-09-15T02:57:52.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-09-15T02:57:52.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-09-15T02:57:52.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-09-15T02:57:52.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-09-15T02:57:52.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-09-15T02:57:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-09-15T02:57:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-09-15T02:57:53.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-09-15T02:57:53.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-09-15T02:57:53.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-15T02:57:53.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-09-15T02:57:53.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-09-15T02:57:53.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-09-15T02:57:53.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-09-15T02:57:53.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-09-15T02:57:53.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-15T02:57:53.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-15T02:57:53.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-09-15T02:57:53.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-09-15T02:57:53.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-09-15T02:57:53.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-09-15T02:57:53.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-09-15T02:57:53.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-09-15T02:57:53.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-09-15T02:57:53.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-09-15T02:57:53.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-09-15T02:57:53.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-09-15T02:57:53.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-09-15T02:57:53.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No data available' 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T02:57:53.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-15T02:57:53.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T02:57:53.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T02:57:53.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-15T02:57:53.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T02:57:53.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-15T02:57:53.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-15T02:57:53.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T02:57:53.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No data available' 2024-09-15T02:57:53.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T02:57:53.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T02:57:53.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-15T02:57:53.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T02:57:53.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T02:57:53.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-15T02:57:53.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T02:57:53.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-09-15T02:57:54.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-09-15T02:57:54.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-15T02:57:54.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T02:57:54.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No such file or directory' 2024-09-15T02:57:54.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T02:57:54.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T02:57:54.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd --striper stat toyfile 2024-09-15T02:57:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T02:57:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T02:57:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-15T02:57:54.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T02:57:54.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-09-15T02:57:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-15T02:57:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T02:57:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No such file or directory' 2024-09-15T02:57:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T02:57:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T02:57:54.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-09-15T02:57:54.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T02:57:54.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T02:57:54.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-15T02:57:54.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T02:57:54.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-09-15T02:57:54.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-15T02:57:54.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:57:54.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-15T02:57:54.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:57:54.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:57:54.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:57:54.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:57:54.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:57:54.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:57:54.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:57:54.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:57:54.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:57:54.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-15T02:57:54.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:57:54.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:54.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:57:54.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71136 2024-09-15T02:57:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:57:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:57:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T02:57:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/rados-striper 0 2024-09-15T02:57:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-15T02:57:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:57:54.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:57:54.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:57:54.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:57:54.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:57:54.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:57:54.224 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:57:54.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:57:54.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:57:54.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:57:54.225 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:57:54.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:57:54.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T02:57:54.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-15T02:57:54.228 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:57:54.228 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:54.228 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71136 2024-09-15T02:57:54.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71136 2024-09-15T02:57:54.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:57:54.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:57:54.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T02:57:54.230 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:57:54.231 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:57:54.295 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-09-15T02:57:54.295 DEBUG:teuthology.orchestra.run.smithi137:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-09-15T02:57:54.359 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:57:54.363 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:57:54.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: run_tests: export .:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:57:54.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2299: export: `.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin': not a valid identifier 2024-09-15T02:57:54.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2301: run_tests: export CEPH_MON=127.0.0.1:7109 2024-09-15T02:57:54.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2301: run_tests: CEPH_MON=127.0.0.1:7109 2024-09-15T02:57:54.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2302: run_tests: export CEPH_ARGS 2024-09-15T02:57:54.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2303: run_tests: uuidgen 2024-09-15T02:57:54.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2303: run_tests: CEPH_ARGS+=' --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none ' 2024-09-15T02:57:54.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2304: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-09-15T02:57:54.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2305: run_tests: export CEPH_CONF=/dev/null 2024-09-15T02:57:54.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2305: run_tests: CEPH_CONF=/dev/null 2024-09-15T02:57:54.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: run_tests: set 2024-09-15T02:57:54.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-09-15T02:57:54.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: run_tests: local 'funcs=test_activate_osd 2024-09-15T02:57:54.367 INFO:tasks.workunit.client.0.smithi137.stderr:test_activate_osd_after_mark_down 2024-09-15T02:57:54.367 INFO:tasks.workunit.client.0.smithi137.stderr:test_activate_osd_skip_benchmark 2024-09-15T02:57:54.367 INFO:tasks.workunit.client.0.smithi137.stderr:test_destroy_osd 2024-09-15T02:57:54.367 INFO:tasks.workunit.client.0.smithi137.stderr:test_display_logs 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_erasure_code_plugin_exists 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_expect_failure 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_flush_pg_stats 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_config 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_is_making_recovery_progress 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_last_scrub_stamp 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_not_primary 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_num_active_clean 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_num_active_or_peered 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_num_pgs 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_op_scheduler 2024-09-15T02:57:54.368 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_osd_id_used_by_pgs 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_osds 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_pg 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_primary 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_get_timeout_delays 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_is_clean 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_kill_daemon 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_kill_daemons 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_objectstore_tool 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_pg_scrub 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_repair 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_run_in_background 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_run_mon 2024-09-15T02:57:54.369 INFO:tasks.workunit.client.0.smithi137.stderr:test_run_osd 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_set_config 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_setup 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_teardown 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_wait_background 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_wait_for_clean 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_wait_for_health_ok 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_wait_for_osd 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_wait_for_peered 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_wait_for_scrub 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:test_wait_osd_id_used_by_pgs' 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: run_tests: local dir=td/ceph-helpers 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T02:57:54.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_activate_osd td/ceph-helpers 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:57:54.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:57:54.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:57:54.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:57:54.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:57:54.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:57:54.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:57:54.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:57:54.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:57:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:57:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:57:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:57:54.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:57:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:57:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:57:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T02:57:54.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:57:54.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:54.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:57:54.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T02:57:54.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:57:54.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:57:54.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T02:57:54.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:57:54.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:54.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:57:54.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:57:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:57:54.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T02:57:54.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T02:57:54.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:57:54.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:57:54.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:57:54.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:57:54.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:54.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:57:54.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:57:54.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:57:54.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:57:54.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:57:54.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:57:54.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T02:57:55.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:57:55.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:57:55.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T02:57:55.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-09-15T02:57:55.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T02:57:55.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:57:55.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:57:55.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:57:55.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T02:57:55.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:57:55.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:57:55.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:57:55.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:57:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:57:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:57:55.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:57:55.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:57:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:57:56.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-09-15T02:57:56.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T02:57:56.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:57:56.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:57:56.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:57:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:57:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T02:57:56.024 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:57:56.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5542735-a68c-479c-bf86-bb2a43e6de69 2024-09-15T02:57:56.025 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 c5542735-a68c-479c-bf86-bb2a43e6de69 2024-09-15T02:57:56.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c5542735-a68c-479c-bf86-bb2a43e6de69' 2024-09-15T02:57:56.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:57:56.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0TeZm3eEkAhAA2kzCA3kUY4WTJDuTpybYEw== 2024-09-15T02:57:56.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0TeZm3eEkAhAA2kzCA3kUY4WTJDuTpybYEw=="}' 2024-09-15T02:57:56.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5542735-a68c-479c-bf86-bb2a43e6de69 -i td/ceph-helpers/0/new.json 2024-09-15T02:57:56.345 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:57:56.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T02:57:56.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0TeZm3eEkAhAA2kzCA3kUY4WTJDuTpybYEw== --osd-uuid c5542735-a68c-479c-bf86-bb2a43e6de69 2024-09-15T02:57:56.379 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:56.379+0000 7fd184020440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:56.399 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:56.399+0000 7fd184020440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:56.400 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:56.400+0000 7fd184020440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:56.400 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:56.400+0000 7fd184020440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:57:56.401 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:56.400+0000 7fd184020440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:57:56.401 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:56.400+0000 7fd184020440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:57:56.439 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:57:56.438+0000 7fd184020440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T02:58:01.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T02:58:01.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:58:01.743 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:58:01.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:58:01.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:58:02.229 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:58:02.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:58:02.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:58:02.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:58:02.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:58:02.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:58:02.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:02.250+0000 7f388b028440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:02.253 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:02.252+0000 7f388b028440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:02.254 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:02.253+0000 7f388b028440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:02.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:58:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:58:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:58:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:58:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:58:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:02.580 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:58:02.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:58:02.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:02.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:02.772 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:02.771+0000 7f388b028440 -1 Falling back to public interface 2024-09-15T02:58:02.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:03.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:03.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:03.932 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:58:03.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:58:03.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:03.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:04.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:05.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:05.249+0000 7f388b028440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:58:05.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:05.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:05.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:58:05.291 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:58:05.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:05.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:05.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:06.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:06.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:06.647 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:58:06.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:58:06.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:06.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:07.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:08.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:08.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:08.057 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:58:08.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:58:08.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:08.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:08.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:09.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:09.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:09.454 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:58:09.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:58:09.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:09.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:09.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:10.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:10.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:10.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:58:10.962 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:58:10.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:10.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:11.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:12.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:12.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:12.332 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:58:12.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:58:12.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:12.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:12.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:13.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:13.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:13.689 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:58:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:58:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:14.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:15.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:15.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:15.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:58:15.118 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:58:15.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:15.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:15.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:16.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:16.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:16.491 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:58:16.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:58:16.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:16.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:17.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:18.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:18.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:18.115 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:58:18.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:58:18.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:18.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:18.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:19.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:19.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:19.757 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:58:19.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:58:19.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:19.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:20.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:21.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:21.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:21.805 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:58:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:58:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:22.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:23.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:23.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:23.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:58:23.556 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:58:23.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:23.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:24.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:25.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:25.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:25.440 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:58:25.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:58:25.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:25.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:26.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:27.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:27.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:27.485 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:58:27.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:58:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:27.954 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/78385,v1:127.0.0.1:6803/78385] [v2:127.0.0.1:6804/78385,v1:127.0.0.1:6805/78385] exists,up c5542735-a68c-479c-bf86-bb2a43e6de69 2024-09-15T02:58:27.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:58:27.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:58:27.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:58:27.956 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-09-15T02:58:27.956 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:58:27.957 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:58:27.957 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:58:27.957 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:27.957 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:27.957 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:58:27.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-09-15T02:58:27.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:58:28.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-15T02:58:28.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-09-15T02:58:28.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-09-15T02:58:28.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-09-15T02:58:28.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:58:28.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:58:28.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:58:28.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:58:28.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:58:28.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:58:28.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-15T02:58:28.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-15T02:58:28.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:58:28.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T02:58:28.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:58:28.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-15T02:58:28.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T02:58:28.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:58:28.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:58:28.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:58:28.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T02:58:28.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T02:58:28.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:58:28.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:58:28.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T02:58:28.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:58:28.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:58:28.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:58:28.351 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:58:28.352 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:28.352 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:28.352 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:58:28.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:58:28.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:58:28.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T02:58:28.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T02:58:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:58:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:58:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:58:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:58:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:58:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-15T02:58:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-15T02:58:28.354 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:58:28.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T02:58:28.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-15T02:58:28.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-15T02:58:28.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T02:58:28.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:58:28.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:58:28.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:58:28.368 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:28.366+0000 7ff7b1123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:28.369 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:28.368+0000 7ff7b1123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:28.371 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:28.369+0000 7ff7b1123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:28.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T02:58:28.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:58:28.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:58:28.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:58:28.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:58:28.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:28.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:58:28.719 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:58:28.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:28.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:28.895 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:28.894+0000 7ff7b1123440 -1 Falling back to public interface 2024-09-15T02:58:29.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:30.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:30.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:30.070 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:58:30.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:58:30.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:30.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:30.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:31.475 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:58:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:58:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:31.798 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:31.797+0000 7ff7b1123440 -1 osd.0 5 log_to_monitors true 2024-09-15T02:58:31.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:32.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:32.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:32.840 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:58:32.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:58:32.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:32.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:33.199 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/79304,v1:127.0.0.1:6803/79304] [v2:127.0.0.1:6804/79304,v1:127.0.0.1:6805/79304] exists,up c5542735-a68c-479c-bf86-bb2a43e6de69 2024-09-15T02:58:33.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:58:33.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:58:33.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:58:33.201 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-09-15T02:58:33.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:58:33.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:58:33.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:58:33.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:58:33.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:58:33.386 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:58:33.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:58:33.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:58:33.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:58:33.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:58:33.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:58:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:58:33.389 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:58:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:58:33.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:58:33.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T02:58:33.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:58:33.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:33.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:33.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T02:58:33.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:58:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:58:33.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:58:33.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:58:33.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:58:33.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:58:33.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:58:33.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:58:33.400 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:58:33.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:58:33.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:58:33.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:58:33.401 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:58:33.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:58:33.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:58:33.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T02:58:33.404 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:58:33.404 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:33.404 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:33.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T02:58:33.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:58:33.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:58:33.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T02:58:33.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:58:33.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:33.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:33.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T02:58:33.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:58:33.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:58:33.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T02:58:33.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T02:58:33.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:58:33.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:58:33.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:58:33.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:58:33.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:33.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:33.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:58:33.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:58:33.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:58:33.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:58:33.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:58:33.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T02:58:34.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:58:34.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:58:34.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T02:58:34.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-09-15T02:58:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T02:58:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:58:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:58:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:58:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T02:58:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:58:34.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:58:34.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:58:34.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:58:34.980 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:58:34.980 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:34.980 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:34.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:58:34.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:58:34.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:58:35.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-09-15T02:58:35.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T02:58:35.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:58:35.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T02:58:35.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T02:58:35.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:58:35.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da73a2ec-b740-42e9-bff4-7d196ab2ce6d 2024-09-15T02:58:35.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 da73a2ec-b740-42e9-bff4-7d196ab2ce6d' 2024-09-15T02:58:35.010 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 da73a2ec-b740-42e9-bff4-7d196ab2ce6d 2024-09-15T02:58:35.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:58:35.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbTeZmOt4/ARAANjUVEc1uOW3SSJ9s7X9LMQ== 2024-09-15T02:58:35.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbTeZmOt4/ARAANjUVEc1uOW3SSJ9s7X9LMQ=="}' 2024-09-15T02:58:35.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da73a2ec-b740-42e9-bff4-7d196ab2ce6d -i td/ceph-helpers/0/new.json 2024-09-15T02:58:35.324 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:58:35.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T02:58:35.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbTeZmOt4/ARAANjUVEc1uOW3SSJ9s7X9LMQ== --osd-uuid da73a2ec-b740-42e9-bff4-7d196ab2ce6d 2024-09-15T02:58:35.358 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:35.358+0000 7facd3a21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:35.370 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:35.370+0000 7facd3a21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:35.371 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:35.371+0000 7facd3a21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:35.371+0000 7facd3a21440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:58:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:35.371+0000 7facd3a21440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:58:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:35.371+0000 7facd3a21440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:58:35.410 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:35.409+0000 7facd3a21440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T02:58:40.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T02:58:40.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:58:40.967 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:58:40.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:58:40.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:58:41.447 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:58:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:58:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:58:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:58:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:58:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:58:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:41.469+0000 7f4300d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:41.471+0000 7f4300d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:41.473 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:41.472+0000 7f4300d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:41.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:58:41.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:58:41.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:58:41.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:58:41.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:58:41.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:41.800 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:58:41.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:58:41.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:41.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:41.986 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:41.986+0000 7f4300d23440 -1 Falling back to public interface 2024-09-15T02:58:42.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:43.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:43.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:43.165 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:58:43.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:58:43.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:43.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:43.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:44.479 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:58:44.478+0000 7f4300d23440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:58:44.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:44.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:44.523 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:58:44.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:58:44.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:44.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:44.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:45.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:45.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:45.943 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:58:45.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:58:45.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:45.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:46.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:47.309 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:58:47.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:58:47.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:47.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:47.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:48.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:48.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:48.698 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:58:48.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:58:48.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:48.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:49.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:50.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:50.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:50.148 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:58:50.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:58:50.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:50.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:50.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:51.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:51.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:51.544 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:58:51.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:58:51.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:51.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:51.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:52.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:52.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:52.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:58:52.903 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:58:52.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:52.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:53.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:54.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:54.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:54.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:58:54.269 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:58:54.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:54.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:54.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:55.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:55.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:55.660 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:58:55.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:58:55.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:55.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:56.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:57.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:57.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:57.043 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:58:57.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:58:57.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:57.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:57.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:58.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:58.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:58.522 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:58:58.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:58:58.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:58.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:58:59.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:00.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:00.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:00.161 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:59:00.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:59:00.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:00.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:00.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:01.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:01.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:01.735 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T02:59:01.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:59:01.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:01.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:02.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:03.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:03.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:03.482 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T02:59:03.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:59:03.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:03.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:04.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:05.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:05.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:05.334 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T02:59:05.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:59:05.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:05.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:06.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:07.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:07.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:07.164 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T02:59:07.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:59:07.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:07.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:07.511 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/80978,v1:127.0.0.1:6803/80978] [v2:127.0.0.1:6804/80978,v1:127.0.0.1:6805/80978] exists,up da73a2ec-b740-42e9-bff4-7d196ab2ce6d 2024-09-15T02:59:07.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:07.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:07.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:07.513 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-15T02:59:07.513 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:59:07.513 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:59:07.513 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:07.514 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:07.514 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:07.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:59:07.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-15T02:59:07.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:59:07.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-09-15T02:59:07.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-09-15T02:59:07.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-09-15T02:59:07.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-09-15T02:59:07.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:07.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:07.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:07.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:07.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:07.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:07.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-09-15T02:59:08.293 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already down. 2024-09-15T02:59:08.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-09-15T02:59:08.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-15T02:59:08.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:59:08.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:08.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:08.305 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:08.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:08.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:08.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:08.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-15T02:59:08.661 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/80978,v1:127.0.0.1:6803/80978] [v2:127.0.0.1:6804/80978,v1:127.0.0.1:6805/80978] exists da73a2ec-b740-42e9-bff4-7d196ab2ce6d 2024-09-15T02:59:08.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:08.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:08.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:08.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:08.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:08.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T02:59:08.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T02:59:08.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:08.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:08.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:08.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:08.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:59:08.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-15T02:59:08.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-15T02:59:08.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T02:59:08.669 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:59:08.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-15T02:59:08.670 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-15T02:59:08.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T02:59:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:59:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:59:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:59:08.686 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:08.684+0000 7faae792f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:08.720 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:08.718+0000 7faae792f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:08.720 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:08.719+0000 7faae792f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:09.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T02:59:09.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:09.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:59:09.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:09.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:09.032 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:09.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:09.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:09.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:09.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:09.237 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:09.236+0000 7faae792f440 -1 Falling back to public interface 2024-09-15T02:59:09.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:10.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:10.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:10.378 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:59:10.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:10.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:10.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:10.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:11.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:11.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:11.785 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:59:11.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:11.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:11.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:12.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:12.144 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:12.142+0000 7faae792f440 -1 osd.0 5 log_to_monitors true 2024-09-15T02:59:13.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:13.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:13.135 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:59:13.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:13.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:13.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:13.508 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/81968,v1:127.0.0.1:6803/81968] [v2:127.0.0.1:6804/81968,v1:127.0.0.1:6805/81968] exists,up da73a2ec-b740-42e9-bff4-7d196ab2ce6d 2024-09-15T02:59:13.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:13.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:13.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:13.509 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-15T02:59:13.510 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:13.589 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:13.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:13.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:13.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:13.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:59:13.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:59:13.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:59:13.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:59:13.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:59:13.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:59:13.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:13.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:59:13.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:59:13.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:13.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:13.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:59:13.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T02:59:13.705 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:59:13.705 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:13.705 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:13.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T02:59:13.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T02:59:13.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:59:13.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T02:59:13.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:13.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:13.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:13.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:13.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:13.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:13.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:59:13.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:59:13.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:59:13.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:59:13.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:59:13.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:59:13.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:13.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:59:13.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:59:13.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:13.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:13.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:59:13.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T02:59:13.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:59:13.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:13.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:13.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T02:59:13.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:59:13.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:59:13.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T02:59:13.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:59:13.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:13.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:13.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T02:59:13.718 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:59:13.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:59:13.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:59:13.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T02:59:13.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-09-15T02:59:13.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T02:59:13.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:59:13.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:59:13.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:59:13.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T02:59:13.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T02:59:14.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:59:14.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:14.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:14.081 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:14.081 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:14.081 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:14.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:14.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:59:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:59:14.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:59:14.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:59:14.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:59:14.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:59:14.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T02:59:14.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:59:14.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:59:14.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:59:14.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:59:14.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:59:14.421 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:59:14.421 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:59:14.421 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:59:14.421 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:14.422 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:14.422 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:14.422 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:59:14.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:59:14.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T02:59:14.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-09-15T02:59:14.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T02:59:14.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:59:14.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:59:14.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:59:14.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T02:59:14.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:59:15.336 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:59:15.336 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:15.336 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:15.337 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:15.337 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:15.337 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:15.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:15.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:59:15.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:59:15.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-15T02:59:15.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T02:59:15.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:59:15.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:59:15.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T02:59:15.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T02:59:15.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T02:59:15.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:59:15.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d0c05f4-cb78-4f9c-84f6-c87328a8d26c 2024-09-15T02:59:15.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0d0c05f4-cb78-4f9c-84f6-c87328a8d26c' 2024-09-15T02:59:15.368 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 0d0c05f4-cb78-4f9c-84f6-c87328a8d26c 2024-09-15T02:59:15.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:59:15.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDTeZm6rmVFhAAMKJEl1Em1ak3PLapOPP2sg== 2024-09-15T02:59:15.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDTeZm6rmVFhAAMKJEl1Em1ak3PLapOPP2sg=="}' 2024-09-15T02:59:15.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d0c05f4-cb78-4f9c-84f6-c87328a8d26c -i td/ceph-helpers/0/new.json 2024-09-15T02:59:15.679 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:15.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T02:59:15.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQCDTeZm6rmVFhAAMKJEl1Em1ak3PLapOPP2sg== --osd-uuid 0d0c05f4-cb78-4f9c-84f6-c87328a8d26c 2024-09-15T02:59:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:15.713+0000 7f8e8a418440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:15.732+0000 7f8e8a418440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:15.733+0000 7f8e8a418440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:15.734+0000 7f8e8a418440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:59:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:15.734+0000 7f8e8a418440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:59:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:15.734+0000 7f8e8a418440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:59:15.774 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:15.772+0000 7f8e8a418440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T02:59:21.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T02:59:21.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:59:21.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:59:21.092 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:59:21.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:59:21.576 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:59:21.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:59:21.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-15T02:59:21.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:59:21.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:59:21.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:59:21.599 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:21.598+0000 7fade3b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:21.601 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:21.600+0000 7fade3b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:21.602 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:21.601+0000 7fade3b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:21.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:59:21.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:21.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:59:21.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:21.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:21.927 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:21.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:21.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:21.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:21.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:22.128 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:22.127+0000 7fade3b21440 -1 Falling back to public interface 2024-09-15T02:59:22.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:23.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:23.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:23.297 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:59:23.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:23.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:23.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:23.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:24.626 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:24.625+0000 7fade3b21440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:59:24.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:24.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:24.690 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:59:24.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:24.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:24.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:25.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:26.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:26.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:26.096 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:59:26.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:26.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:26.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:26.453 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/83642,v1:127.0.0.1:6803/83642] [v2:127.0.0.1:6804/83642,v1:127.0.0.1:6805/83642] exists,up 0d0c05f4-cb78-4f9c-84f6-c87328a8d26c 2024-09-15T02:59:26.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:26.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:26.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:26.454 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-15T02:59:26.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-15T02:59:26.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-15T02:59:26.533 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-15T02:59:26.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-15T02:59:26.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-15T02:59:26.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-09-15T02:59:26.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:26.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:26.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:26.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:26.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:26.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:26.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-09-15T02:59:27.296 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already down. 2024-09-15T02:59:27.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-09-15T02:59:27.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-15T02:59:27.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:59:27.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:27.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:27.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:27.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:27.308 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:27.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:27.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-15T02:59:27.656 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/83642,v1:127.0.0.1:6803/83642] [v2:127.0.0.1:6804/83642,v1:127.0.0.1:6805/83642] exists 0d0c05f4-cb78-4f9c-84f6-c87328a8d26c 2024-09-15T02:59:27.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:27.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:27.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:27.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-15T02:59:27.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-15T02:59:27.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:59:27.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T02:59:27.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:27.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:27.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:27.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:59:27.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-15T02:59:27.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-15T02:59:27.663 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:59:27.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T02:59:27.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-15T02:59:27.664 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-15T02:59:27.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T02:59:27.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:59:27.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:59:27.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:59:27.684 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:27.680+0000 7f859e432440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:27.701 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:27.699+0000 7f859e432440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:27.702 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:27.700+0000 7f859e432440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:28.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T02:59:28.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:28.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:59:28.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:28.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:28.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:28.022 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:28.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:28.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:28.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:28.217 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:28.215+0000 7f859e432440 -1 Falling back to public interface 2024-09-15T02:59:28.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:29.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:29.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:29.371 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:59:29.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:29.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:29.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:29.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:30.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:30.733 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:59:30.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:30.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:30.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:30.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:31.008 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:31.007+0000 7f859e432440 -1 osd.0 6 log_to_monitors true 2024-09-15T02:59:31.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:32.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:32.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:32.101 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:59:32.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:32.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:32.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:32.453 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/84283,v1:127.0.0.1:6803/84283] [v2:127.0.0.1:6804/84283,v1:127.0.0.1:6805/84283] exists,up 0d0c05f4-cb78-4f9c-84f6-c87328a8d26c 2024-09-15T02:59:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-15T02:59:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:59:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:59:32.457 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:32.457 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:32.457 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:32.457 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:59:32.457 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-15T02:59:32.457 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-15T02:59:32.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-15T02:59:32.536 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-15T02:59:32.536 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:59:32.536 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:59:32.536 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:32.536 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:32.537 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:32.537 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T02:59:32.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-15T02:59:32.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T02:59:32.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:32.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:32.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:32.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:32.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:32.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:32.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:59:32.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:59:32.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:59:32.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:59:32.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:59:32.721 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:59:32.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:32.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:59:32.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:59:32.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:32.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:32.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:59:32.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T02:59:32.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:59:32.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:32.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:32.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T02:59:32.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:59:32.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:59:32.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T02:59:32.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_destroy_osd td/ceph-helpers 2024-09-15T02:59:32.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-09-15T02:59:32.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:32.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:32.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:32.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:32.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:32.733 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:59:32.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:59:32.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:59:32.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:59:32.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:59:32.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:59:32.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:32.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:59:32.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:59:32.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:59:32.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:59:32.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:59:32.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T02:59:32.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:59:32.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:32.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:32.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T02:59:32.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:59:32.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:59:32.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T02:59:32.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:59:32.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:32.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:32.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T02:59:32.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:59:32.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T02:59:32.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T02:59:33.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:59:33.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:33.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:33.094 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:33.094 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:33.094 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:33.095 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:33.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:59:33.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:59:33.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:59:33.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T02:59:33.429 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:59:33.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T02:59:33.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-09-15T02:59:33.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T02:59:33.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:59:33.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:59:33.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:59:33.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T02:59:33.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:34.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:59:34.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:59:34.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-09-15T02:59:34.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T02:59:34.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:59:34.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:59:34.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:59:34.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T02:59:34.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T02:59:34.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:59:34.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=053c11bf-193f-42b4-b8ac-9b7b56b93398 2024-09-15T02:59:34.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 053c11bf-193f-42b4-b8ac-9b7b56b93398' 2024-09-15T02:59:34.373 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 053c11bf-193f-42b4-b8ac-9b7b56b93398 2024-09-15T02:59:34.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:59:34.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWTeZmFJfeFhAAs17qyBnj8DJAGY5Oq3f+ng== 2024-09-15T02:59:34.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWTeZmFJfeFhAAs17qyBnj8DJAGY5Oq3f+ng=="}' 2024-09-15T02:59:34.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 053c11bf-193f-42b4-b8ac-9b7b56b93398 -i td/ceph-helpers/0/new.json 2024-09-15T02:59:34.676 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:34.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T02:59:34.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWTeZmFJfeFhAAs17qyBnj8DJAGY5Oq3f+ng== --osd-uuid 053c11bf-193f-42b4-b8ac-9b7b56b93398 2024-09-15T02:59:34.710 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:34.710+0000 7f7939b82440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:34.721 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:34.721+0000 7f7939b82440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:34.722 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:34.722+0000 7f7939b82440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:34.723 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:34.722+0000 7f7939b82440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:59:34.723 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:34.722+0000 7f7939b82440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:59:34.723 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:34.722+0000 7f7939b82440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T02:59:34.762 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:34.761+0000 7f7939b82440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T02:59:40.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T02:59:40.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:59:40.192 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T02:59:40.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:59:40.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:59:40.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:59:40.672 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T02:59:40.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:59:40.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:59:40.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:59:40.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:59:40.694 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:40.693+0000 7f83e3d88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:40.696 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:40.695+0000 7f83e3d88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:40.697 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:40.696+0000 7f83e3d88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:41.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:59:41.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:41.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:59:41.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:41.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:41.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:41.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:41.021 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T02:59:41.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:41.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:41.225 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:41.225+0000 7f83e3d88440 -1 Falling back to public interface 2024-09-15T02:59:41.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:42.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:42.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:42.376 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T02:59:42.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:42.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:42.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:42.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:43.722 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T02:59:43.721+0000 7f83e3d88440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:59:43.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:43.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:43.738 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T02:59:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:44.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:45.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:45.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:45.101 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T02:59:45.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:45.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:45.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:45.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:46.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:46.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:46.493 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T02:59:46.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:59:46.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:46.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:46.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:47.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:47.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:47.858 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T02:59:47.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:59:47.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:47.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:48.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:49.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:49.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:49.218 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T02:59:49.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:59:49.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:49.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:49.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:50.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:50.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:50.711 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T02:59:50.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:59:50.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:50.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:51.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:52.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:52.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:52.117 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T02:59:52.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:59:52.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:52.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:52.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:53.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:53.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:53.523 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T02:59:53.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:59:53.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:53.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:53.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:54.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:54.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:54.925 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T02:59:54.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:59:54.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:54.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:55.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:56.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:56.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:56.289 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T02:59:56.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:59:56.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:56.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:56.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:57.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:57.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:57.751 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T02:59:57.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:59:57.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:57.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:59:58.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:59.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:59:59.799 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T02:59:59.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:59.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:00.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:01.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:01.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:01.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:00:01.613 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:00:01.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:01.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:02.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:03.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:03.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:03.294 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:00:03.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:00:03.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:03.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:04.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:05.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:05.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:05.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:00:05.096 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:00:05.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:05.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:05.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:06.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:06.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:06.931 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:00:06.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:00:06.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:06.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:07.278 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85960,v1:127.0.0.1:6803/85960] [v2:127.0.0.1:6804/85960,v1:127.0.0.1:6805/85960] exists,up 053c11bf-193f-42b4-b8ac-9b7b56b93398 2024-09-15T03:00:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:00:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:00:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:00:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-09-15T03:00:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-09-15T03:00:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-09-15T03:00:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-09-15T03:00:07.803 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already out. 2024-09-15T03:00:07.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-15T03:00:07.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:07.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:07.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:07.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:07.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:08.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:08.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-09-15T03:00:08.514 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already down. 2024-09-15T03:00:08.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-09-15T03:00:08.940 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 does not exist 2024-09-15T03:00:08.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-09-15T03:00:08.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-09-15T03:00:08.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:08.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-09-15T03:00:08.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:08.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:08.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:08.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:08.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:08.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:08.953 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:08.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:08.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:08.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:08.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:08.956 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:08.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:08.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:08.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:08.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:08.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:08.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:08.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-09-15T03:00:08.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:08.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:08.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:08.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:08.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:08.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:08.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-09-15T03:00:08.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-09-15T03:00:08.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-09-15T03:00:09.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-09-15T03:00:09.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:09.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:09.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:09.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:09.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:09.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:09.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:09.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:09.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:09.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:09.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:09.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:09.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:09.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:09.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:09.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:09.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:09.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:09.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:09.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:09.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:09.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:09.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:09.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:09.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:09.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_display_logs td/ceph-helpers 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: test_display_logs: local dir=td/ceph-helpers 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: test_display_logs: setup td/ceph-helpers 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:00:09.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:09.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:09.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:09.427 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:09.427 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:09.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:09.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:09.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:09.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:09.428 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:09.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:09.428 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:09.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:09.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:09.429 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:09.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:09.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:09.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:09.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:09.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:09.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:09.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:09.433 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:09.433 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:09.433 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:09.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:09.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:09.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:09.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:00:09.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:00:09.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:09.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:09.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:00:09.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:00:09.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:00:09.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:00:09.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:00:09.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: test_display_logs: run_mon td/ceph-helpers a 2024-09-15T03:00:09.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:00:09.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:00:09.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:00:09.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:00:09.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:00:09.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T03:00:09.798 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:00:09.798 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:09.798 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:09.798 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:09.798 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:09.798 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:09.799 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:09.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:00:10.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:00:10.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:00:10.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:10.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:10.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:10.051 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:10.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:10.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:10.128 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:10.129 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:10.129 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:10.129 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:10.129 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:10.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:10.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:00:10.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_display_logs: kill_daemons td/ceph-helpers 2024-09-15T03:00:10.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:10.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:10.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:10.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:10.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:11.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:11.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: test_display_logs: display_logs td/ceph-helpers 2024-09-15T03:00:11.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: display_logs: local dir=td/ceph-helpers 2024-09-15T03:00:11.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-09-15T03:00:11.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: display_logs: read file 2024-09-15T03:00:11.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-09-15T03:00:11.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: display_logs: cat td/ceph-helpers/mon.a.log 2024-09-15T03:00:11.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: display_logs: read file 2024-09-15T03:00:11.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-09-15T03:00:11.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: test_display_logs: teardown td/ceph-helpers 2024-09-15T03:00:11.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:11.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:11.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:11.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:11.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:11.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:11.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:11.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:11.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:11.518 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:11.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:11.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:11.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:11.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:11.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:11.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:11.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:11.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:11.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:11.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:11.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:11.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-09-15T03:00:11.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2044: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:11.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:11.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:11.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:11.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:11.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:11.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:11.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:11.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:11.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:11.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:11.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:11.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:11.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:11.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:11.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:11.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:11.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:00:11.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:00:11.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:11.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:11.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:00:11.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:00:11.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:00:11.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:00:11.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:00:11.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-09-15T03:00:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:00:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:00:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:00:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:00:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:00:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T03:00:11.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:00:11.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:11.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:11.914 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:11.914 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:11.914 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:11.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:11.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:00:12.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:00:12.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:00:12.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:12.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:12.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:00:12.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:00:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:12.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:12.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:12.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:12.252 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:12.252 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:12.252 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:12.252 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:12.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:12.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:00:12.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-09-15T03:00:12.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:00:12.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:00:12.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:00:12.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:00:12.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:00:12.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:00:13.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:13.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:13.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-09-15T03:00:13.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: erasure_code_plugin_exists: local plugin=jerasure 2024-09-15T03:00:13.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: erasure_code_plugin_exists: local status 2024-09-15T03:00:13.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: local grepstr 2024-09-15T03:00:13.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: erasure_code_plugin_exists: local s 2024-09-15T03:00:13.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: case `uname` in 2024-09-15T03:00:13.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: uname 2024-09-15T03:00:13.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-09-15T03:00:13.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-09-15T03:00:13.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: s= 2024-09-15T03:00:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: erasure_code_plugin_exists: local status=0 2024-09-15T03:00:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-15T03:00:13.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-15T03:00:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-15T03:00:13.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: erasure_code_plugin_exists: return 0 2024-09-15T03:00:13.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-09-15T03:00:13.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: erasure_code_plugin_exists: local plugin=FAKE 2024-09-15T03:00:13.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: erasure_code_plugin_exists: local status 2024-09-15T03:00:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: local grepstr 2024-09-15T03:00:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: erasure_code_plugin_exists: local s 2024-09-15T03:00:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: case `uname` in 2024-09-15T03:00:13.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: uname 2024-09-15T03:00:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-09-15T03:00:13.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-09-15T03:00:14.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-09-15T03:00:14.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: erasure_code_plugin_exists: local status=5 2024-09-15T03:00:14.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-09-15T03:00:14.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-09-15T03:00:14.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-09-15T03:00:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: erasure_code_plugin_exists: return 5 2024-09-15T03:00:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-09-15T03:00:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:14.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:14.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:14.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:14.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:14.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:14.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:14.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:14.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:14.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:14.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:14.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:14.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:14.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:14.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:14.146 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:14.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:14.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:14.149 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:14.149 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.149 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:14.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:14.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:14.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_expect_failure td/ceph-helpers 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: test_expect_failure: local dir=td/ceph-helpers 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: test_expect_failure: setup td/ceph-helpers 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:14.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:14.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:14.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:14.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:14.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:14.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:14.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:14.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:14.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:14.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:14.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:14.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:14.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:14.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:14.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:14.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:14.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:00:14.161 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:00:14.161 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.161 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/ceph-helpers 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=FAIL 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T03:00:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T03:00:14.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-09-15T03:00:14.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T03:00:14.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T03:00:14.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-15T03:00:14.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T03:00:14.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1959: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-09-15T03:00:14.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/ceph-helpers 2024-09-15T03:00:14.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T03:00:14.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=FAIL 2024-09-15T03:00:14.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T03:00:14.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T03:00:14.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-09-15T03:00:14.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1940: expect_failure: success=true 2024-09-15T03:00:14.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: true 2024-09-15T03:00:14.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: expect_failure: cat td/ceph-helpers/out 2024-09-15T03:00:14.167 INFO:tasks.workunit.client.0.smithi137.stderr:FAIL 2024-09-15T03:00:14.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: expect_failure: return 1 2024-09-15T03:00:14.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1960: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-15T03:00:14.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-09-15T03:00:14.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/ceph-helpers 2024-09-15T03:00:14.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T03:00:14.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=FAIL 2024-09-15T03:00:14.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T03:00:14.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T03:00:14.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-09-15T03:00:14.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T03:00:14.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T03:00:14.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-15T03:00:14.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: expect_failure: cat td/ceph-helpers/out 2024-09-15T03:00:14.171 INFO:tasks.workunit.client.0.smithi137.stderr:UNEXPECTED 2024-09-15T03:00:14.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: expect_failure: return 1 2024-09-15T03:00:14.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-15T03:00:14.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: test_expect_failure: teardown td/ceph-helpers 2024-09-15T03:00:14.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:14.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:14.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:14.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:14.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:14.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:14.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:14.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:14.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:14.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:14.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:14.175 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:14.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:14.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:14.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:14.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:14.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:14.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:14.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:14.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:14.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:14.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_flush_pg_stats td/ceph-helpers 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2196: test_flush_pg_stats: local dir=td/ceph-helpers 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2198: test_flush_pg_stats: setup td/ceph-helpers 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:00:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:14.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:14.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:14.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:14.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:14.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:14.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:14.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:14.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:14.190 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:14.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:14.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:14.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:14.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:00:14.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:00:14.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:00:14.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:00:14.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:00:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:14.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:00:14.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:00:14.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:00:14.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:14.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:14.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:14.801 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:00:14.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:00:14.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:14.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:14.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:00:14.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-09-15T03:00:14.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:00:14.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:00:14.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:00:14.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:00:14.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:00:14.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:00:15.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:00:15.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:15.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:15.786 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:15.786 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:15.786 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:15.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:15.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:15.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:15.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-09-15T03:00:15.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:00:15.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:00:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:00:15.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:00:15.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d2324921-4ba9-4274-b840-c88a0f6afa7d 2024-09-15T03:00:15.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d2324921-4ba9-4274-b840-c88a0f6afa7d' 2024-09-15T03:00:15.816 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 d2324921-4ba9-4274-b840-c88a0f6afa7d 2024-09-15T03:00:15.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:00:15.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/TeZmw0xJMRAANyDcGtLea4K7JW1t4c3Sng== 2024-09-15T03:00:15.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/TeZmw0xJMRAANyDcGtLea4K7JW1t4c3Sng=="}' 2024-09-15T03:00:15.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d2324921-4ba9-4274-b840-c88a0f6afa7d -i td/ceph-helpers/0/new.json 2024-09-15T03:00:16.162 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:00:16.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:00:16.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/TeZmw0xJMRAANyDcGtLea4K7JW1t4c3Sng== --osd-uuid d2324921-4ba9-4274-b840-c88a0f6afa7d 2024-09-15T03:00:16.197 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:16.195+0000 7f73f6123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:16.209 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:16.207+0000 7f73f6123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:16.210 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:16.208+0000 7f73f6123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:16.210 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:16.209+0000 7f73f6123440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:00:16.210 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:16.209+0000 7f73f6123440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:00:16.210 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:16.209+0000 7f73f6123440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:00:16.249 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:16.247+0000 7f73f6123440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:00:21.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:00:21.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:00:21.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:00:21.625 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:00:21.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:00:22.104 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:00:22.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:00:22.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:00:22.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:00:22.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:00:22.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:00:22.126 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:22.125+0000 7ff723b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:22.128 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:22.127+0000 7ff723b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:22.129 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:22.128+0000 7ff723b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:22.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:00:22.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:00:22.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:00:22.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:00:22.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:00:22.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:22.512 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:00:22.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:00:22.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:22.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:22.644 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:22.643+0000 7ff723b23440 -1 Falling back to public interface 2024-09-15T03:00:22.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:23.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:23.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:23.861 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:00:23.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:00:23.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:23.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:24.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:25.119 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:25.117+0000 7ff723b23440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:00:25.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:25.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:25.214 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:00:25.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:00:25.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:25.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:25.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:26.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:26.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:26.779 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:00:26.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:00:26.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:26.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:27.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:28.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:28.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:28.191 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:00:28.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:00:28.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:28.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:29.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:29.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:29.596 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:00:29.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:00:29.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:29.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:30.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:31.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:31.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:31.019 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:00:31.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:00:31.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:31.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:31.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:32.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:32.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:32.383 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:00:32.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:00:32.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:32.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:32.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:33.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:33.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:33.932 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:00:33.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:00:33.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:33.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:34.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:35.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:35.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:35.296 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:00:35.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:00:35.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:35.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:35.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:36.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:36.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:36.658 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:00:36.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:00:36.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:36.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:37.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:38.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:38.690 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:00:38.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:38.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:00:38.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:38.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:39.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:40.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:40.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:40.306 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:00:40.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:00:40.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:40.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:42.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:42.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:42.012 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:00:42.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:00:42.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:42.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:43.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:44.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:44.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:44.150 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:00:44.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:00:44.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:44.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:45.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:46.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:46.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:46.020 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:00:46.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:00:46.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:46.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:46.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:47.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:47.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:47.588 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:00:47.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:00:47.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:47.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:00:47.997 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/88505,v1:127.0.0.1:6803/88505] [v2:127.0.0.1:6804/88505,v1:127.0.0.1:6805/88505] exists,up d2324921-4ba9-4274-b840-c88a0f6afa7d 2024-09-15T03:00:47.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:00:47.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:00:47.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:00:47.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: test_flush_pg_stats: create_rbd_pool 2024-09-15T03:00:47.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:00:48.433 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:00:48.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:00:48.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:00:48.921 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:00:48.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:00:49.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:00:51.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-09-15T03:00:51.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2204: test_flush_pg_stats: flush_pg_stats 2024-09-15T03:00:51.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:00:51.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:00:51.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:00:51.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:00:51.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:51.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:00:51.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-15T03:00:51.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-15T03:00:51.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-15T03:00:51.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:51.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-15T03:00:51.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:51.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:00:51.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-15T03:00:51.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:51.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-15T03:00:51.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-15T03:00:51.714 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836485 2024-09-15T03:00:51.715 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-15T03:00:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:00:53.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:00:53.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:53.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-15T03:00:53.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-09-15T03:00:53.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_flush_pg_stats: ceph df detail --format=json 2024-09-15T03:00:53.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-09-15T03:00:53.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_flush_pg_stats: stored=6798 2024-09-15T03:00:53.912 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: test_flush_pg_stats: ceph df detail --format=json 2024-09-15T03:00:53.912 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-09-15T03:00:54.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: test_flush_pg_stats: stored_raw=6798 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: test_flush_pg_stats: test 6798 -gt 0 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: test_flush_pg_stats: test 6798 == 6798 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: test_flush_pg_stats: teardown td/ceph-helpers 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:54.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:54.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:54.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:54.485 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:54.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:54.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:54.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:54.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:54.487 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:54.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:54.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:54.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:54.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:54.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:54.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:54.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:54.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:54.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:54.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:54.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_config td/ceph-helpers 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:00:54.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:54.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:54.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:54.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:54.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:54.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:54.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:54.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:54.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:54.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:54.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:54.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:54.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:54.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:54.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:54.502 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:54.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:54.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:54.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:00:54.505 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:54.505 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:54.505 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:54.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:00:54.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:54.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:54.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:00:54.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:00:54.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:54.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:54.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:00:54.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:00:54.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:00:54.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:00:54.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:00:54.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:00:54.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:00:54.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:00:54.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:00:54.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:00:54.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:00:54.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:00:54.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:00:54.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:54.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:54.864 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:54.864 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:54.864 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:54.864 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:54.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:00:55.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:00:55.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:00:55.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:55.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:55.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:55.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:00:55.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:00:55.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:55.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:55.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:00:55.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:00:55.199 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:55.199 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:55.200 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:55.200 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:55.200 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:55.200 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:55.200 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:55.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:55.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:00:55.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-09-15T03:00:55.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:55.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:55.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-09-15T03:00:55.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-09-15T03:00:55.274 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:55.274 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:55.275 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:55.275 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:55.275 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:55.275 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:55.275 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:00:55.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:55.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get osd_pool_default_size 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:00:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:00:56.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:00:56.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:56.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:56.193 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:56.193 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:56.194 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:56.194 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:56.194 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:56.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:00:56.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:56.222 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:00:56.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:00:56.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-09-15T03:00:56.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:00:56.224 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:00:56.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11dd8044-8874-4e10-9306-43c70e13ef11 2024-09-15T03:00:56.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 11dd8044-8874-4e10-9306-43c70e13ef11' 2024-09-15T03:00:56.225 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 11dd8044-8874-4e10-9306-43c70e13ef11 2024-09-15T03:00:56.225 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:00:56.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoTeZmshoJDhAAwfGP2naWUCY2taRtSmlUCQ== 2024-09-15T03:00:56.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoTeZmshoJDhAAwfGP2naWUCY2taRtSmlUCQ=="}' 2024-09-15T03:00:56.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11dd8044-8874-4e10-9306-43c70e13ef11 -i td/ceph-helpers/0/new.json 2024-09-15T03:00:56.536 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:00:56.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:00:56.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQDoTeZmshoJDhAAwfGP2naWUCY2taRtSmlUCQ== --osd-uuid 11dd8044-8874-4e10-9306-43c70e13ef11 2024-09-15T03:00:56.571 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:56.570+0000 7ff838123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:56.582 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:56.581+0000 7ff838123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:56.583 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:56.582+0000 7ff838123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:56.584 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:56.582+0000 7ff838123440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:00:56.584 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:56.582+0000 7ff838123440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:00:56.584 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:56.582+0000 7ff838123440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:00:56.623 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:00:56.621+0000 7ff838123440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:01:01.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:01:01.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:01:01.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:01:01.917 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:01:01.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:01:02.420 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:01:02.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:01:02.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-09-15T03:01:02.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:01:02.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:01:02.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:01:02.454 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:02.452+0000 7f5d1e32c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:02.472 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:02.470+0000 7f5d1e32c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:02.472 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:02.471+0000 7f5d1e32c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:02.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:01:02.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:01:02.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:01:02.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:01:02.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:01:02.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:02.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:01:02.816 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:01:02.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:02.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:03.013 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:03.011+0000 7f5d1e32c440 -1 Falling back to public interface 2024-09-15T03:01:03.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:04.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:04.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:04.173 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:01:04.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:01:04.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:04.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:04.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:05.483 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:05.481+0000 7f5d1e32c440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:01:05.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:05.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:05.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:01:05.522 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:01:05.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:05.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:05.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:06.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:06.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:06.883 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:01:06.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:01:06.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:06.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:07.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:08.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:08.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:08.355 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:01:08.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:01:08.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:08.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:08.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:09.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:09.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:09.729 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:01:09.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:01:09.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:09.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:11.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:11.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:11.144 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:01:11.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:01:11.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:11.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:11.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:12.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:12.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:12.507 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:01:12.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:01:12.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:12.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:12.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:13.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:13.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:13.912 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:01:13.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:01:13.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:13.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:14.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:15.302 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:01:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:01:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:15.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:16.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:16.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:16.687 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:01:16.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:01:16.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:16.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:17.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:18.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:18.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:18.215 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:01:18.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:01:18.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:18.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:18.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:19.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:19.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:19.653 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:01:19.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:01:19.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:19.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:20.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:21.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:21.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:21.686 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:01:21.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:01:21.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:21.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:22.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:23.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:23.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:23.632 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:01:23.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:01:23.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:23.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:24.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:25.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:25.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:25.370 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:01:25.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:01:25.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:25.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:25.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:26.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:27.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:27.000 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:01:27.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:01:27.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:27.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:27.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:28.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:28.484 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:01:28.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:28.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:28.873 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/90764,v1:127.0.0.1:6803/90764] [v2:127.0.0.1:6804/90764,v1:127.0.0.1:6805/90764] exists,up 11dd8044-8874-4e10-9306-43c70e13ef11 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-09-15T03:01:28.876 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-15T03:01:28.877 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:01:28.877 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:01:28.877 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:01:28.877 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:28.878 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:28.878 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T03:01:28.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:01:28.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_max_scrubs 2024-09-15T03:01:28.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-09-15T03:01:28.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-09-15T03:01:28.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:01:28.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:01:28.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:01:28.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:01:28.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:01:28.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:01:28.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:01:28.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:01:29.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:01:29.060 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:01:29.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:01:29.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:01:29.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:01:29.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:01:29.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:01:29.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:29.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:01:29.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:01:29.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:29.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:01:29.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:01:29.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:01:29.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:01:29.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:29.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:29.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:01:29.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:01:29.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:01:29.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:01:29.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:01:29.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:01:29.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:01:29.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:01:29.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:01:29.075 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:01:29.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:01:29.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:01:29.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:01:29.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:29.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:01:29.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:01:29.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:29.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:01:29.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:01:29.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:01:29.080 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:01:29.080 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:29.080 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:29.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:01:29.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:01:29.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:01:29.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:01:29.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:01:29.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:29.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:29.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:01:29.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:01:29.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:01:29.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:01:29.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:01:29.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:01:29.680 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:01:29.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:29.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:29.681 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:01:29.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:01:29.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:01:29.754 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:01:29.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:01:29.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:01:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-09-15T03:01:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:01:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:01:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:01:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:01:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:01:29.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:01:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:01:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:01:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:01:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:01:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:01:30.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:01:30.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:01:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-09-15T03:01:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:01:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:01:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:01:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:01:30.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:01:30.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:01:30.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:01:30.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:01:30.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-09-15T03:01:30.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:01:30.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:01:30.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:01:30.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:01:30.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:01:30.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:01:30.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:01:30.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:01:31.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:01:31.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:01:31.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:01:31.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:01:31.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:01:31.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:01:31.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:01:31.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:31.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:01:31.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:01:31.004 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:31.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:01:31.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:01:31.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:01:31.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:01:31.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:31.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:31.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:01:31.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:01:31.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:01:31.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:01:31.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-09-15T03:01:31.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-09-15T03:01:31.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-09-15T03:01:31.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:01:31.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:01:31.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:01:31.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:01:31.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:01:31.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:01:31.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:01:31.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:01:31.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:01:31.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:01:31.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:01:31.018 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:01:31.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:01:31.018 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:01:31.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:01:31.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:01:31.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:01:31.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:31.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:01:31.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:01:31.020 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:01:31.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:01:31.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:01:31.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:01:31.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:01:31.026 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:31.026 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:31.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:01:31.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:01:31.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:01:31.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:01:31.027 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:01:31.027 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:31.027 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:31.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:01:31.028 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:01:31.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:01:31.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:01:31.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:01:31.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:01:31.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:01:31.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:01:31.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:01:31.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:01:31.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:01:31.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:01:31.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:01:31.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:01:31.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:01:31.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:01:31.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:31.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:31.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:01:31.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:01:31.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:01:31.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:01:31.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:01:31.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:01:31.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:01:31.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:01:31.629 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:01:31.629 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:01:31.630 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:01:31.630 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:01:31.630 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:31.630 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:31.630 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:01:31.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:01:31.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:01:31.705 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:01:31.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:01:31.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:01:31.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-09-15T03:01:31.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:01:31.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:01:31.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:01:31.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:01:31.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:01:31.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:01:32.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:01:32.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:01:32.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:01:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:01:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:32.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:01:32.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:01:32.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:01:32.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:01:32.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:01:32.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:01:32.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:01:32.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57eed780-b825-41e0-add3-2a1f2c589f0b 2024-09-15T03:01:32.645 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 57eed780-b825-41e0-add3-2a1f2c589f0b 2024-09-15T03:01:32.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 57eed780-b825-41e0-add3-2a1f2c589f0b' 2024-09-15T03:01:32.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:01:32.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMTuZmGRceJxAASrJzfULi0xaN5A73/ZamSQ== 2024-09-15T03:01:32.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMTuZmGRceJxAASrJzfULi0xaN5A73/ZamSQ=="}' 2024-09-15T03:01:32.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57eed780-b825-41e0-add3-2a1f2c589f0b -i td/ceph-helpers/0/new.json 2024-09-15T03:01:32.959 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:01:32.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:01:32.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMTuZmGRceJxAASrJzfULi0xaN5A73/ZamSQ== --osd-uuid 57eed780-b825-41e0-add3-2a1f2c589f0b 2024-09-15T03:01:32.993 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:32.993+0000 7f4e8cd7b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:33.004 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:33.004+0000 7f4e8cd7b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:33.005 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:33.005+0000 7f4e8cd7b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:33.006 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:33.005+0000 7f4e8cd7b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:01:33.006 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:33.005+0000 7f4e8cd7b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:01:33.006 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:33.005+0000 7f4e8cd7b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:01:33.044 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:33.044+0000 7f4e8cd7b440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:01:38.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:01:38.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:01:38.412 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:01:38.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:01:38.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:01:38.898 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:01:38.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:01:38.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:01:38.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:01:38.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:01:38.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:01:38.921 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:38.920+0000 7f8853f4e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:38.922 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:38.922+0000 7f8853f4e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:38.923 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:38.923+0000 7f8853f4e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:39.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:01:39.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:01:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:01:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:01:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:01:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:01:39.247 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:01:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:39.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:39.437 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:39.437+0000 7f8853f4e440 -1 Falling back to public interface 2024-09-15T03:01:39.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:40.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:40.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:40.600 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:01:40.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:01:40.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:40.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:40.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:41.947 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:01:41.945+0000 7f8853f4e440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:01:41.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:41.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:41.959 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:01:41.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:01:41.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:41.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:42.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:43.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:43.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:43.339 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:01:43.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:01:43.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:43.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:43.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:44.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:44.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:44.794 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:01:44.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:01:44.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:44.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:45.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:46.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:46.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:46.182 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:01:46.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:01:46.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:46.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:46.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:47.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:47.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:47.631 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:01:47.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:01:47.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:47.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:48.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:49.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:49.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:49.010 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:01:49.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:01:49.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:49.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:49.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:50.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:50.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:50.391 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:01:50.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:01:50.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:50.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:50.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:51.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:51.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:51.779 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:01:51.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:01:51.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:51.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:52.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:53.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:53.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:53.139 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:01:53.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:01:53.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:53.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:53.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:54.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:54.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:54.592 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:01:54.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:01:54.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:54.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:56.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:56.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:56.328 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:01:56.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:01:56.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:56.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:57.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:58.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:58.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:58.490 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:01:58.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:01:58.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:58.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:59.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:00.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:00.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:00.209 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:02:00.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:02:00.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:00.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:01.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:02.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:02.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:02.053 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:02:02.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:02:02.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:02.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:03.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:04.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:04.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:04.213 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:02:04.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:02:04.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:04.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:04.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:05.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:05.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:05.677 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:02:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:02:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:06.035 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/92990,v1:127.0.0.1:6803/92990] [v2:127.0.0.1:6804/92990,v1:127.0.0.1:6805/92990] exists,up 57eed780-b825-41e0-add3-2a1f2c589f0b 2024-09-15T03:02:06.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:02:06.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:02:06.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:02:06.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-09-15T03:02:06.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:02:06.406 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:02:06.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:02:06.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:02:06.939 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:02:06.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:02:07.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:02:09.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-09-15T03:02:09.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:02:09.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:02:09.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:02:09.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:02:09.144 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:02:09.144 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:02:09.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:02:09.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:02:09.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:02:09.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:02:09.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:02:09.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:02:09.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:02:09.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:02:09.240 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:02:09.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:02:09.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:02:09.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:02:09.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:02:09.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-15T03:02:09.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-15T03:02:09.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-15T03:02:09.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:02:09.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-15T03:02:09.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:02:09.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:02:09.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-15T03:02:09.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:02:09.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-15T03:02:09.694 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836492 2024-09-15T03:02:09.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-15T03:02:09.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:02:10.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-15T03:02:10.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:02:11.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:02:11.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:02:11.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-15T03:02:11.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:02:11.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:02:11.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:02:11.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:02:11.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:02:11.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:02:11.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:02:11.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:02:11.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:02:11.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:02:11.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:02:12.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:02:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:02:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:02:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:02:12.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:02:13.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-09-15T03:02:06.677462+0000 2024-09-15T03:02:13.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-09-15T03:02:06.677462+0000 2024-09-15T03:02:13.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-09-15T03:02:13.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:02:13.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:02:13.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:02:13.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:02:13.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:02:13.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:02:13.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:02:13.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:02:13.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:02:13.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:02:13.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:02:13.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:02:13.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:02:13.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:02:13.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:02:13.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:02:13.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:02:13.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:02:13.147 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:02:13.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:02:13.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:02:13.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:02:13.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:02:13.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:13.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:13.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:02:13.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:02:13.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:02:13.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_not_primary td/ceph-helpers 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:02:13.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:02:13.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:02:13.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:02:13.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:02:13.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:02:13.158 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:02:13.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:02:13.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:02:13.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:02:13.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:02:13.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:02:13.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:02:13.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:02:13.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:02:13.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:02:13.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:02:13.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:02:13.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:13.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:13.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:02:13.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:02:13.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:02:13.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:02:13.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:02:13.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:13.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:13.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:02:13.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:02:13.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:02:13.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:02:13.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:02:13.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T03:02:13.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:13.759 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:13.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:02:13.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:02:13.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:02:13.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:02:13.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:02:13.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:02:13.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:02:13.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:02:13.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:02:13.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:02:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:02:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:02:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:02:13.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:02:13.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:02:13.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-09-15T03:02:13.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:02:13.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:02:13.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:02:13.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:02:13.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:02:13.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:02:14.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:02:14.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:02:14.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:02:14.744 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:02:14.744 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:14.745 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:14.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:02:14.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:02:14.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:02:14.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:14.773 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:02:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:02:14.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:02:14.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:02:14.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8432b8fb-24e0-4c48-9e00-2ea76fc535b5 2024-09-15T03:02:14.775 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 8432b8fb-24e0-4c48-9e00-2ea76fc535b5 2024-09-15T03:02:14.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8432b8fb-24e0-4c48-9e00-2ea76fc535b5' 2024-09-15T03:02:14.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:02:14.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2TuZmcp3mLhAA8eEm4vU4+VxP2/bHLEczlA== 2024-09-15T03:02:14.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2TuZmcp3mLhAA8eEm4vU4+VxP2/bHLEczlA=="}' 2024-09-15T03:02:14.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8432b8fb-24e0-4c48-9e00-2ea76fc535b5 -i td/ceph-helpers/0/new.json 2024-09-15T03:02:15.096 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:02:15.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:02:15.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2TuZmcp3mLhAA8eEm4vU4+VxP2/bHLEczlA== --osd-uuid 8432b8fb-24e0-4c48-9e00-2ea76fc535b5 2024-09-15T03:02:15.131 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:15.129+0000 7fbd7d97f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:15.142 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:15.141+0000 7fbd7d97f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:15.143 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:15.142+0000 7fbd7d97f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:15.143 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:15.142+0000 7fbd7d97f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:02:15.143 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:15.142+0000 7fbd7d97f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:02:15.144 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:15.142+0000 7fbd7d97f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:02:15.182 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:15.180+0000 7fbd7d97f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:02:20.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:02:20.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:02:20.488 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:02:20.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:02:20.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:02:20.978 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:02:20.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:02:20.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:02:20.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:02:20.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:02:20.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:02:21.001 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:21.000+0000 7f268051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:21.003 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:21.003+0000 7f268051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:21.004 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:21.004+0000 7f268051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:21.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:02:21.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:02:21.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:02:21.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:02:21.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:02:21.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:21.326 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:02:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:02:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:21.535 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:21.535+0000 7f268051f440 -1 Falling back to public interface 2024-09-15T03:02:21.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:22.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:22.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:22.696 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:02:22.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:02:22.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:22.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:23.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:24.042 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:24.042+0000 7f268051f440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:02:24.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:24.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:24.086 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:02:24.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:02:24.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:24.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:24.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:25.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:25.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:25.612 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:02:25.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:02:25.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:25.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:26.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:27.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:27.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:27.045 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:02:27.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:02:27.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:27.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:27.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:28.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:28.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:28.429 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:02:28.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:02:28.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:28.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:28.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:29.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:29.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:29.888 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:02:29.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:02:29.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:29.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:30.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:31.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:31.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:31.283 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:02:31.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:02:31.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:31.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:31.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:32.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:32.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:32.691 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:02:32.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:02:32.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:32.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:33.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:34.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:34.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:34.078 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:02:34.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:02:34.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:34.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:34.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:35.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:35.455 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:02:35.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:35.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:02:35.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:35.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:36.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:37.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:37.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:37.188 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:02:37.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:02:37.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:37.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:37.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:38.903 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:02:38.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:02:38.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:38.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:39.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:40.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:40.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:40.640 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:02:40.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:02:40.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:40.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:41.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:42.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:42.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:42.730 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:02:42.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:02:42.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:42.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:43.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:44.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:44.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:44.691 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:02:44.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:02:44.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:44.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:45.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:46.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:46.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:46.593 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:02:46.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:02:46.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:46.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:02:46.958 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/95328,v1:127.0.0.1:6803/95328] [v2:127.0.0.1:6804/95328,v1:127.0.0.1:6805/95328] exists,up 8432b8fb-24e0-4c48-9e00-2ea76fc535b5 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:02:46.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:02:46.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:02:46.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:02:46.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:02:46.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:02:46.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:02:46.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:46.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:02:46.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:02:46.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:02:46.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:02:46.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:02:46.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:02:46.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:02:46.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=620a4b84-6b33-4faa-a1f1-4400a19d1162 2024-09-15T03:02:46.967 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 620a4b84-6b33-4faa-a1f1-4400a19d1162 2024-09-15T03:02:46.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 620a4b84-6b33-4faa-a1f1-4400a19d1162' 2024-09-15T03:02:46.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:02:46.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWTuZmNIkmOhAAbUFZH14pNxM7Qb5Q5YSJaw== 2024-09-15T03:02:46.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWTuZmNIkmOhAAbUFZH14pNxM7Qb5Q5YSJaw=="}' 2024-09-15T03:02:46.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 620a4b84-6b33-4faa-a1f1-4400a19d1162 -i td/ceph-helpers/1/new.json 2024-09-15T03:02:47.480 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:02:47.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:02:47.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWTuZmNIkmOhAAbUFZH14pNxM7Qb5Q5YSJaw== --osd-uuid 620a4b84-6b33-4faa-a1f1-4400a19d1162 2024-09-15T03:02:47.516 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:47.515+0000 7fa4d652f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:47.518 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:47.517+0000 7fa4d652f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:47.519 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:47.518+0000 7fa4d652f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:47.519 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:47.518+0000 7fa4d652f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:02:47.520 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:47.518+0000 7fa4d652f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:02:47.520 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:47.518+0000 7fa4d652f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:02:47.596 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:47.595+0000 7fa4d652f440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:02:53.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:02:53.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:02:53.705 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:02:53.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:02:53.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:02:54.182 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:02:54.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:02:54.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:02:54.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:02:54.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:02:54.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:02:54.204 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:54.203+0000 7f0557615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:54.206 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:54.205+0000 7f0557615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:54.207 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:54.206+0000 7f0557615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:54.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:02:54.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:02:54.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:02:54.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:02:54.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:02:54.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:54.574 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:02:54.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:02:54.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:54.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:02:54.724 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:54.723+0000 7f0557615440 -1 Falling back to public interface 2024-09-15T03:02:54.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:55.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:55.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:55.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:02:55.934 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:02:55.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:55.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:02:56.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:57.289 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:02:57.287+0000 7f0557615440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:02:57.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:57.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:57.319 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:02:57.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:02:57.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:57.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:02:57.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:58.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:58.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:58.951 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:02:58.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:02:58.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:58.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:02:59.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:00.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:00.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:00.496 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:03:00.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:03:00.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:00.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:00.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:01.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:01.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:01.858 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:03:01.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:03:01.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:01.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:02.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:03.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:03.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:03.349 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:03:03.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:03:03.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:03.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:03.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:04.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:04.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:04.747 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:03:04.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:03:04.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:04.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:05.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:06.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:06.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:06.253 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:03:06.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:03:06.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:06.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:06.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:07.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:07.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:07.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:03:07.650 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:03:07.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:07.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:08.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:09.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:09.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:09.090 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:03:09.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:03:09.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:09.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:09.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:10.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:10.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:10.452 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:03:10.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:03:10.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:10.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:11.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:12.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:12.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:12.616 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:03:12.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:03:12.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:12.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:13.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:14.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:14.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:14.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:03:14.447 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:03:14.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:14.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:15.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:16.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:16.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:16.208 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:03:16.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:03:16.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:16.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:16.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:17.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:17.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:17.977 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:03:17.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:03:17.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:17.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:19.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:19.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:19.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:03:19.733 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:03:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:20.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:21.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:21.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:21.090 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:03:21.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:03:21.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:21.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:21.444 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/97057,v1:127.0.0.1:6811/97057] [v2:127.0.0.1:6812/97057,v1:127.0.0.1:6813/97057] exists,up 620a4b84-6b33-4faa-a1f1-4400a19d1162 2024-09-15T03:03:21.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:03:21.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:03:21.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:03:21.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-09-15T03:03:21.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:03:21.800 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:03:21.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:03:21.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:03:22.676 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:03:22.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:03:23.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:03:25.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-09-15T03:03:25.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:03:25.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:03:25.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:03:25.109 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:03:25.109 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:03:25.110 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:03:25.110 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:03:25.110 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:03:25.110 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:03:25.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:03:25.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:03:25.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:03:25.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:03:25.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:03:25.208 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:03:25.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:03:25.553 INFO:tasks.workunit.client.0.smithi137.stderr:1' 2024-09-15T03:03:25.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:03:25.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:25.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:03:25.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-15T03:03:25.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-15T03:03:25.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-15T03:03:25.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:25.660 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:03:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672968 2024-09-15T03:03:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672968 2024-09-15T03:03:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672968' 2024-09-15T03:03:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:25.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-15T03:03:25.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:25.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:03:25.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-15T03:03:25.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:25.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-15T03:03:25.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-15T03:03:25.777 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836492 2024-09-15T03:03:25.777 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:03:26.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-15T03:03:26.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:03:27.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:03:27.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:03:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-15T03:03:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:27.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672968 2024-09-15T03:03:27.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:27.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:03:27.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672968 2024-09-15T03:03:27.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:27.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672968 2024-09-15T03:03:27.489 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672968 2024-09-15T03:03:27.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-15T03:03:27.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:03:27.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672968 -lt 42949672968 2024-09-15T03:03:27.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:03:27.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:27.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:28.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:03:28.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:03:28.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:03:28.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:03:28.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:03:28.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:03:28.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:03:28.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:03:28.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:03:28.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:03:28.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:28.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:29.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:03:29.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:03:29.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:03:29.182 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-09-15T03:03:29.182 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-15T03:03:29.182 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-15T03:03:29.182 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-15T03:03:29.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T03:03:29.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-09-15T03:03:29.552 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-09-15T03:03:29.552 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-09-15T03:03:29.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-09-15T03:03:29.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-09-15T03:03:29.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-15T03:03:29.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-15T03:03:29.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-15T03:03:29.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T03:03:29.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T03:03:29.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-09-15T03:03:29.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:03:30.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:03:30.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:03:30.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:03:30.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:03:30.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:03:30.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:03:30.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:03:30.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:03:30.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:03:30.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:03:30.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:03:30.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:03:30.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:03:30.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:03:30.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:03:30.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:03:30.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:03:30.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:03:30.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:30.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:30.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:03:30.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:03:30.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:03:30.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:03:30.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_num_active_clean td/ceph-helpers 2024-09-15T03:03:30.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-09-15T03:03:30.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:03:30.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:03:30.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:03:30.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:03:30.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:03:30.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:03:30.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:03:30.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:03:30.428 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:03:30.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:03:30.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:03:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:03:30.429 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:03:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:03:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:03:30.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:03:30.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:03:30.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:30.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:30.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:03:30.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:03:30.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:03:30.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:03:30.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:03:30.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:30.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:30.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:03:30.434 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:03:30.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:03:30.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:03:30.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:03:30.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:03:30.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:03:30.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:03:30.794 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:03:30.794 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:30.794 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:30.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:03:30.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:03:31.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:03:31.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:03:31.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:03:31.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:03:31.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:03:31.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:03:31.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:03:31.142 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:03:31.142 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:03:31.142 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:03:31.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:03:31.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-09-15T03:03:31.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:03:31.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:03:31.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:03:31.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:03:31.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:03:31.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:03:32.058 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:03:32.058 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:03:32.058 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:03:32.058 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:03:32.058 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:32.058 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:32.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:03:32.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:03:32.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:03:32.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:03:32.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:32.087 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:03:32.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f16989b6-0390-4f9e-8087-37279d0674dc 2024-09-15T03:03:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f16989b6-0390-4f9e-8087-37279d0674dc' 2024-09-15T03:03:32.090 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 f16989b6-0390-4f9e-8087-37279d0674dc 2024-09-15T03:03:32.090 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:03:32.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCETuZmMDXxBRAAqA5Pz2v4VJ9VSOy/nFdU6w== 2024-09-15T03:03:32.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCETuZmMDXxBRAAqA5Pz2v4VJ9VSOy/nFdU6w=="}' 2024-09-15T03:03:32.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f16989b6-0390-4f9e-8087-37279d0674dc -i td/ceph-helpers/0/new.json 2024-09-15T03:03:32.408 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:03:32.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:03:32.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCETuZmMDXxBRAAqA5Pz2v4VJ9VSOy/nFdU6w== --osd-uuid f16989b6-0390-4f9e-8087-37279d0674dc 2024-09-15T03:03:32.442 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:32.442+0000 7efd38321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:32.454+0000 7efd38321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:32.455+0000 7efd38321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:32.455+0000 7efd38321440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:03:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:32.455+0000 7efd38321440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:03:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:32.455+0000 7efd38321440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:03:32.495 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:32.494+0000 7efd38321440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:03:37.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:03:37.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:03:37.854 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:03:37.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:03:37.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:03:38.335 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:03:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:03:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:03:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:03:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:03:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:03:38.358 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:38.356+0000 7f8579c40440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:38.360 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:38.359+0000 7f8579c40440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:38.361 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:38.360+0000 7f8579c40440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:03:38.686 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:03:38.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:38.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:38.884 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:38.883+0000 7f8579c40440 -1 Falling back to public interface 2024-09-15T03:03:39.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:40.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:40.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:40.050 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:03:40.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:03:40.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:40.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:40.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:41.380 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:03:41.378+0000 7f8579c40440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:03:41.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:41.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:41.409 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:03:41.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:03:41.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:41.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:41.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:42.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:42.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:42.780 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:03:42.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:03:42.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:42.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:43.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:44.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:44.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:44.230 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:03:44.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:03:44.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:44.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:44.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:45.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:45.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:45.631 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:03:45.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:03:45.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:45.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:46.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:47.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:47.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:47.074 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:03:47.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:03:47.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:47.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:47.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:48.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:48.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:48.445 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:03:48.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:03:48.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:48.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:48.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:49.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:49.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:49.808 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:03:49.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:03:49.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:49.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:50.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:51.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:51.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:51.198 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:03:51.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:03:51.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:51.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:51.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:52.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:52.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:52.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:03:52.587 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:03:52.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:52.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:53.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:54.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:54.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:54.023 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:03:54.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:03:54.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:54.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:54.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:55.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:55.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:55.448 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:03:55.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:03:55.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:55.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:55.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:56.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:56.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:56.871 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:03:56.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:03:56.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:56.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:57.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:58.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:58.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:58.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:03:58.892 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:03:58.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:58.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:03:59.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:00.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:00.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:00.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:04:00.536 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:04:00.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:00.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:01.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:02.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:02.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:02.603 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:04:02.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:04:02.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:02.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:03.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:04.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:04.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:04.175 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:04:04.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:04:04.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:04.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:04.530 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/99503,v1:127.0.0.1:6803/99503] [v2:127.0.0.1:6804/99503,v1:127.0.0.1:6805/99503] exists,up f16989b6-0390-4f9e-8087-37279d0674dc 2024-09-15T03:04:04.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:04:04.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:04:04.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:04:04.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-09-15T03:04:04.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:04:04.894 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:04:04.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:04:04.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:04:05.415 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:04:05.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:04:06.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:04:07.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-09-15T03:04:07.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:04:07.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:04:07.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:04:07.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:04:07.399 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:04:07.399 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:04:07.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:04:07.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:04:07.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:04:07.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:04:07.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:04:07.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:04:07.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:04:07.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:04:07.493 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:04:07.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:04:07.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:04:07.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:07.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-15T03:04:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-15T03:04:07.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-15T03:04:07.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:07.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-15T03:04:07.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:07.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:04:07.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-15T03:04:07.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:07.964 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836486 2024-09-15T03:04:07.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-15T03:04:07.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-15T03:04:07.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:08.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-15T03:04:08.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:04:09.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:04:09.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:09.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-15T03:04:09.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:04:09.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:09.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:10.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:10.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:04:10.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:04:10.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:10.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:11.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:04:11.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:04:11.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:04:11.045 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-09-15T03:04:11.045 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:11.046 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:11.046 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:11.046 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:11.046 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:11.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-09-15T03:04:11.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-09-15T03:04:11.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-09-15T03:04:11.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:04:11.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:04:11.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:04:11.433 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:11.434 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:11.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:11.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:11.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:11.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:11.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:04:11.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:04:11.539 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:04:11.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:04:11.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:04:11.541 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:04:11.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:04:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:04:11.543 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:04:11.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:04:11.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:04:11.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:04:11.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:11.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:11.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:04:11.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:04:11.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:04:11.555 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:04:11.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:04:11.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:04:11.557 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:04:11.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:11.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:04:11.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:04:11.557 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:11.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:04:11.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:04:11.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:04:11.558 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:04:11.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:11.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:11.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:04:11.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:04:11.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:04:11.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:04:11.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:04:11.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:11.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:11.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:04:11.562 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:04:11.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:04:11.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:04:11.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:04:11.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:04:11.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:04:11.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:04:11.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:04:11.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:04:11.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:04:11.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:04:11.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:04:11.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:11.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:11.910 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:11.910 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:11.910 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:11.911 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:11.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:04:12.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:12.164 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:12.165 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:04:12.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:04:12.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:04:12.238 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:04:12.239 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:12.239 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:12.239 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:04:12.239 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:04:12.239 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:04:12.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-09-15T03:04:12.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:04:12.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:04:12.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:04:12.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:04:12.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:04:12.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:04:13.147 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:04:13.147 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:13.147 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:13.147 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:13.147 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:13.148 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:13.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:13.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:04:13.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:04:13.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-09-15T03:04:13.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:04:13.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:04:13.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:04:13.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:04:13.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:04:13.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:04:13.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:13.176 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:04:13.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:04:13.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:04:13.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:04:13.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:04:13.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac533155-1bd4-406a-8e9d-0b6dfb0ec63f 2024-09-15T03:04:13.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ac533155-1bd4-406a-8e9d-0b6dfb0ec63f' 2024-09-15T03:04:13.178 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 ac533155-1bd4-406a-8e9d-0b6dfb0ec63f 2024-09-15T03:04:13.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:04:13.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtTuZmjdg6CxAAg7pfVoWyxPPNjhAZSQVqEg== 2024-09-15T03:04:13.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtTuZmjdg6CxAAg7pfVoWyxPPNjhAZSQVqEg=="}' 2024-09-15T03:04:13.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac533155-1bd4-406a-8e9d-0b6dfb0ec63f -i td/ceph-helpers/0/new.json 2024-09-15T03:04:13.533 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:04:13.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:04:13.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtTuZmjdg6CxAAg7pfVoWyxPPNjhAZSQVqEg== --osd-uuid ac533155-1bd4-406a-8e9d-0b6dfb0ec63f 2024-09-15T03:04:13.568 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:13.567+0000 7ff861422440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:13.578+0000 7ff861422440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:13.579+0000 7ff861422440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:13.580+0000 7ff861422440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:04:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:13.580+0000 7ff861422440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:04:13.581 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:13.580+0000 7ff861422440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:04:13.619 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:13.619+0000 7ff861422440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:04:19.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:04:19.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:04:19.220 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:04:19.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:04:19.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:04:19.705 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:04:19.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:04:19.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:04:19.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:04:19.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:04:19.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:04:19.727 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:19.726+0000 7fd5ef435440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:19.729 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:19.729+0000 7fd5ef435440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:19.730 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:19.730+0000 7fd5ef435440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:20.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:04:20.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:04:20.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:04:20.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:04:20.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:04:20.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:20.098 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:04:20.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:04:20.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:20.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:20.256 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:20.256+0000 7fd5ef435440 -1 Falling back to public interface 2024-09-15T03:04:20.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:21.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:21.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:21.454 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:04:21.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:04:21.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:21.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:21.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:22.738 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:22.738+0000 7fd5ef435440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:04:22.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:22.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:22.812 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:04:22.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:04:22.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:22.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:23.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:24.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:24.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:24.349 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:04:24.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:04:24.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:24.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:24.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:25.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:25.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:25.811 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:04:25.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:04:25.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:25.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:26.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:27.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:27.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:27.167 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:04:27.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:04:27.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:27.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:27.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:28.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:28.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:28.606 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:04:28.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:04:28.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:28.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:28.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:29.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:29.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:29.987 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:04:29.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:04:29.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:29.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:30.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:31.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:31.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:31.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:04:31.428 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:04:31.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:31.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:31.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:32.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:32.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:32.789 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:04:32.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:04:32.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:32.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:33.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:34.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:34.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:34.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:04:34.151 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:04:34.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:34.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:35.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:36.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:36.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:36.075 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:04:36.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:04:36.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:36.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:36.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:37.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:37.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:37.632 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:04:37.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:04:37.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:37.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:38.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:39.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:39.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:39.294 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:04:39.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:04:39.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:39.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:40.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:41.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:41.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:41.385 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:04:41.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:04:41.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:41.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:42.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:43.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:43.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:43.221 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:04:43.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:04:43.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:43.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:44.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:45.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:45.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:45.275 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:04:45.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:04:45.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:45.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:04:45.645 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/101842,v1:127.0.0.1:6803/101842] [v2:127.0.0.1:6804/101842,v1:127.0.0.1:6805/101842] exists,up ac533155-1bd4-406a-8e9d-0b6dfb0ec63f 2024-09-15T03:04:45.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:04:45.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:04:45.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:04:45.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-09-15T03:04:45.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:04:45.985 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:04:45.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:04:45.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:04:46.599 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:04:46.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:04:47.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:04:48.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-09-15T03:04:48.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:04:48.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:04:48.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:04:48.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:04:48.647 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:04:48.647 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:04:48.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:04:48.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:04:48.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:04:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:04:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:04:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:04:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:04:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:04:48.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:04:49.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:04:49.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:04:49.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:49.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836488 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836488 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488' 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836488 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:04:49.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836488 2024-09-15T03:04:49.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:49.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836488 2024-09-15T03:04:49.206 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836488 2024-09-15T03:04:49.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-15T03:04:49.207 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:49.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-15T03:04:49.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:04:50.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:04:50.556 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:50.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836488 2024-09-15T03:04:50.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:04:50.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:50.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:51.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:04:51.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:04:51.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:04:51.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:51.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:51.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:51.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:51.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:51.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:04:51.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:04:51.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:51.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-15T03:04:52.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:52.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:52.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:52.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:52.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:52.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:52.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:04:52.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:04:52.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:04:52.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:04:52.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:04:52.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:04:52.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:52.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:04:52.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:04:52.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:52.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:04:52.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:04:52.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:04:52.767 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:04:52.767 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:52.768 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:52.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:04:52.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:04:52.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_num_pgs td/ceph-helpers 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:04:52.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:04:52.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:52.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:52.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:52.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:52.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:52.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:52.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:04:52.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:04:52.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:04:52.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:04:52.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:04:52.774 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:04:52.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:52.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:04:52.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:04:52.774 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:04:52.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:04:52.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:04:52.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:04:52.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:04:52.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:52.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:52.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:04:52.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:04:52.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:04:52.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:04:52.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:04:52.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:04:52.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:04:52.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:04:52.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:04:52.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:53.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:04:53.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:04:53.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:04:53.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:04:53.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:04:53.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:04:53.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:04:53.357 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:04:53.357 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:04:53.358 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:04:53.358 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:04:53.358 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:53.358 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:53.358 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:04:53.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:04:53.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:04:53.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:04:53.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:04:53.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:04:53.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:04:53.433 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:04:53.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-09-15T03:04:53.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:04:53.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:04:53.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:04:53.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:04:53.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:04:53.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:04:54.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:04:54.350 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:54.350 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:54.350 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:54.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:04:54.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:04:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-09-15T03:04:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:04:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:04:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:04:54.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:04:54.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:04:54.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:04:54.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:04:54.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:04:54.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:04:54.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:04:54.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:04:54.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:04:54.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:04:54.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e850cb9f-6bf5-474b-8d83-489808e6d370 2024-09-15T03:04:54.385 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 e850cb9f-6bf5-474b-8d83-489808e6d370 2024-09-15T03:04:54.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e850cb9f-6bf5-474b-8d83-489808e6d370' 2024-09-15T03:04:54.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:04:54.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWTuZmHyFUFxAAT/unpqaADYPLWVldcZN79w== 2024-09-15T03:04:54.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWTuZmHyFUFxAAT/unpqaADYPLWVldcZN79w=="}' 2024-09-15T03:04:54.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e850cb9f-6bf5-474b-8d83-489808e6d370 -i td/ceph-helpers/0/new.json 2024-09-15T03:04:54.694 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:04:54.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:04:54.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWTuZmHyFUFxAAT/unpqaADYPLWVldcZN79w== --osd-uuid e850cb9f-6bf5-474b-8d83-489808e6d370 2024-09-15T03:04:54.729 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:54.727+0000 7f517591f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:54.740 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:54.739+0000 7f517591f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:54.740+0000 7f517591f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:54.740+0000 7f517591f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:04:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:54.740+0000 7f517591f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:04:54.746 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:54.740+0000 7f517591f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:04:54.772 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:04:54.771+0000 7f517591f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:05:00.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:05:00.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:05:00.141 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:05:00.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:05:00.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:05:00.627 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:05:00.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:05:00.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:05:00.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:05:00.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:05:00.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:05:00.650 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:00.648+0000 7f309c35e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:00.652 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:00.650+0000 7f309c35e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:00.653 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:00.652+0000 7f309c35e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:00.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:05:00.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:05:00.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:05:00.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:05:00.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:05:00.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:00.980 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:05:00.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:05:00.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:00.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:01.166 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:01.165+0000 7f309c35e440 -1 Falling back to public interface 2024-09-15T03:05:01.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:02.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:02.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:02.344 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:05:02.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:05:02.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:02.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:02.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:03.641 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:03.639+0000 7f309c35e440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:05:03.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:03.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:03.700 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:05:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:05:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:04.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:05.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:05.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:05.210 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:05:05.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:05:05.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:05.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:05.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:06.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:06.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:06.648 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:05:06.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:05:06.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:06.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:07.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:08.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:08.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:08.005 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:05:08.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:05:08.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:08.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:08.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:09.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:09.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:09.443 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:05:09.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:05:09.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:09.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:09.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:10.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:10.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:10.873 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:05:10.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:05:10.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:10.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:11.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:12.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:12.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:12.331 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:05:12.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:05:12.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:12.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:12.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:13.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:13.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:13.727 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:05:13.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:05:13.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:13.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:14.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:15.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:15.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:15.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:05:15.124 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:05:15.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:15.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:15.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:16.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:16.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:16.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:05:16.920 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:05:16.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:16.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:17.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:18.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:18.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:18.467 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:05:18.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:05:18.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:18.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:19.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:20.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:20.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:20.137 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:05:20.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:05:20.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:20.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:20.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:21.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:21.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:21.844 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:05:21.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:05:21.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:21.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:22.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:23.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:23.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:23.425 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:05:23.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:05:23.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:23.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:24.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:25.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:25.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:25.082 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:05:25.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:05:25.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:25.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:25.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:26.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:26.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:26.637 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:05:26.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:05:26.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:26.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:26.993 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/104155,v1:127.0.0.1:6803/104155] [v2:127.0.0.1:6804/104155,v1:127.0.0.1:6805/104155] exists,up e850cb9f-6bf5-474b-8d83-489808e6d370 2024-09-15T03:05:26.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:05:26.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:05:26.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:05:26.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-09-15T03:05:26.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:05:27.334 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:05:27.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:05:27.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:05:27.961 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:05:27.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:05:28.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:05:30.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-09-15T03:05:30.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:05:30.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:05:30.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:05:30.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:05:30.062 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:05:30.062 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:05:30.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:05:30.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:05:30.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:05:30.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:05:30.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:05:30.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:05:30.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:05:30.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:05:30.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:05:30.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:05:30.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:05:30.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:05:30.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:05:30.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836484 2024-09-15T03:05:30.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836484 2024-09-15T03:05:30.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484' 2024-09-15T03:05:30.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:05:30.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836484 2024-09-15T03:05:30.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:05:30.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:05:30.637 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836484 2024-09-15T03:05:30.637 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:05:30.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836484 2024-09-15T03:05:30.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-15T03:05:30.638 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836484 2024-09-15T03:05:30.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:05:30.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-15T03:05:30.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:05:31.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:05:31.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:05:32.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-15T03:05:32.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:05:32.340 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:05:32.340 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:05:32.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:05:32.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:05:32.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:05:32.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:05:32.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:05:32.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:05:32.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:05:32.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:05:33.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:05:33.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:05:33.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:05:33.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:05:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:05:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:05:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:05:33.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-09-15T03:05:33.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:05:33.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:05:34.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:05:34.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:05:34.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:05:34.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:05:34.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:05:34.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:05:34.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:05:34.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:05:34.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:05:34.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:34.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:05:34.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:05:34.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:34.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:05:34.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:05:34.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:05:34.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:05:34.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:34.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:34.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:05:34.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:05:34.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_op_scheduler td/ceph-helpers 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: test_get_op_scheduler: local dir=td/ceph-helpers 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: test_get_op_scheduler: setup td/ceph-helpers 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:05:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:05:34.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:05:34.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:05:34.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:05:34.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:05:34.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:05:34.264 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:05:34.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:05:34.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:05:34.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:05:34.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:34.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:05:34.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:05:34.266 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:34.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:05:34.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:05:34.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:05:34.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:05:34.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:34.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:34.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:05:34.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:05:34.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:05:34.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:05:34.271 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:05:34.271 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:34.271 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:34.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:05:34.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:05:34.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:05:34.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T03:05:34.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:05:34.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:34.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:34.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:34.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:34.624 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:34.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:05:34.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:05:34.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:05:34.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:05:34.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:05:34.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:05:34.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:05:34.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:05:34.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:05:34.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:05:34.960 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:05:34.960 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:05:34.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:05:34.961 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:05:34.961 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:34.961 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:34.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:05:34.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:05:34.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:05:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-09-15T03:05:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:05:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:05:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:05:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:05:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:05:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:05:35.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:05:35.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:35.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:35.877 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:35.877 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:35.877 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:35.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:05:35.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:05:35.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:05:35.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-09-15T03:05:35.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:05:35.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:05:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:05:35.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:05:35.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:05:35.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3615a236-df1c-4490-8f18-44ddc1f3cff8 2024-09-15T03:05:35.908 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 3615a236-df1c-4490-8f18-44ddc1f3cff8 2024-09-15T03:05:35.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3615a236-df1c-4490-8f18-44ddc1f3cff8' 2024-09-15T03:05:35.908 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:05:35.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/TuZmUXG/NhAApy9XT0MqPqENAJZMUrcSJA== 2024-09-15T03:05:35.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/TuZmUXG/NhAApy9XT0MqPqENAJZMUrcSJA=="}' 2024-09-15T03:05:35.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3615a236-df1c-4490-8f18-44ddc1f3cff8 -i td/ceph-helpers/0/new.json 2024-09-15T03:05:36.222 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:05:36.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:05:36.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQD/TuZmUXG/NhAApy9XT0MqPqENAJZMUrcSJA== --osd-uuid 3615a236-df1c-4490-8f18-44ddc1f3cff8 2024-09-15T03:05:36.257 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:36.255+0000 7f215a816440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:36.275 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:36.274+0000 7f215a816440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:36.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:36.275+0000 7f215a816440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:36.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:36.275+0000 7f215a816440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:05:36.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:36.275+0000 7f215a816440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:05:36.278 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:36.275+0000 7f215a816440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:05:36.307 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:36.305+0000 7f215a816440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:05:41.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:05:41.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:05:41.905 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:05:41.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:05:41.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:05:42.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:05:42.386 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:05:42.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-09-15T03:05:42.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:05:42.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:05:42.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:05:42.410 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:42.408+0000 7ff908411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:42.412 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:42.410+0000 7ff908411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:42.412 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:42.411+0000 7ff908411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:42.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:05:42.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:05:42.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:05:42.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:05:42.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:05:42.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:42.745 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:05:42.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:05:42.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:42.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:42.933 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:42.932+0000 7ff908411440 -1 Falling back to public interface 2024-09-15T03:05:43.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:44.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:44.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:44.104 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:05:44.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:05:44.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:44.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:44.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:45.435 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:45.433+0000 7ff908411440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:05:45.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:45.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:45.460 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:05:45.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:05:45.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:45.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:45.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:46.903 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:05:46.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:05:46.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:46.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:47.252 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/106493,v1:127.0.0.1:6803/106493] [v2:127.0.0.1:6804/106493,v1:127.0.0.1:6805/106493] exists,up 3615a236-df1c-4490-8f18-44ddc1f3cff8 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_get_op_scheduler: get_op_scheduler 0 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: get_op_scheduler: local id=0 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: get_op_scheduler: get_config osd 0 osd_op_queue 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-15T03:05:47.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:05:47.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_op_queue 2024-09-15T03:05:47.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_get_op_scheduler: test wpq = wpq 2024-09-15T03:05:47.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-09-15T03:05:47.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:05:47.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:05:47.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:05:47.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:47.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:05:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:05:47.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57d97048-098a-4a7a-9bf7-cd569bc6287f 2024-09-15T03:05:47.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 57d97048-098a-4a7a-9bf7-cd569bc6287f' 2024-09-15T03:05:47.341 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 57d97048-098a-4a7a-9bf7-cd569bc6287f 2024-09-15T03:05:47.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:05:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALT+ZmyFn5FBAAfxMD6g1quj1LADIIxbNtmQ== 2024-09-15T03:05:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALT+ZmyFn5FBAAfxMD6g1quj1LADIIxbNtmQ=="}' 2024-09-15T03:05:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57d97048-098a-4a7a-9bf7-cd569bc6287f -i td/ceph-helpers/1/new.json 2024-09-15T03:05:47.831 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:05:47.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:05:47.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQALT+ZmyFn5FBAAfxMD6g1quj1LADIIxbNtmQ== --osd-uuid 57d97048-098a-4a7a-9bf7-cd569bc6287f 2024-09-15T03:05:47.868 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:47.866+0000 7f23eba8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:47.869 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:47.868+0000 7f23eba8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:47.870 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:47.869+0000 7f23eba8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:47.871 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:47.869+0000 7f23eba8d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:05:47.871 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:47.869+0000 7f23eba8d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:05:47.871 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:47.869+0000 7f23eba8d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:05:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:47.942+0000 7f23eba8d440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:05:53.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:05:53.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:05:53.515 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:05:53.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:05:53.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:05:53.998 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:05:53.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:05:53.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-15T03:05:53.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:05:53.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:05:53.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:05:54.022 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:54.020+0000 7f5701213440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:54.023 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:54.022+0000 7f5701213440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:54.024 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:54.023+0000 7f5701213440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:05:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:05:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:05:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:05:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:05:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:54.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:05:54.351 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:05:54.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:54.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:05:54.538 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:54.537+0000 7f5701213440 -1 Falling back to public interface 2024-09-15T03:05:54.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:55.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:55.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:55.710 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:05:55.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:05:55.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:55.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:05:56.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:57.014 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:05:57.012+0000 7f5701213440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:05:57.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:57.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:57.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:05:57.098 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:05:57.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:57.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:05:57.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:58.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:58.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:58.756 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:05:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:05:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:05:59.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:00.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:00.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:00.125 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:06:00.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:06:00.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:00.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:00.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:01.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:01.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:01.545 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:06:01.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:06:01.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:01.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:02.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:03.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:03.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:03.076 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:06:03.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:06:03.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:03.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:03.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:04.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:04.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:04.474 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:06:04.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:06:04.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:04.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:04.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:05.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:05.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:05.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:06:05.954 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:06:05.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:05.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:06.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:07.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:07.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:07.463 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:06:07.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:06:07.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:07.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:07.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:08.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:08.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:08.884 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:06:08.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:06:08.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:08.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:09.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:10.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:10.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:10.249 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:06:10.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:06:10.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:10.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:11.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:12.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:12.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:12.197 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:06:12.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:06:12.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:12.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:12.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:13.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:13.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:13.926 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:06:13.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:06:13.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:13.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:14.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:15.735 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:06:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:06:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:16.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:17.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:17.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:17.453 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:06:17.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:06:17.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:17.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:17.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:18.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:18.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:18.948 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:06:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:06:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:19.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:20.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:20.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:20.460 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:06:20.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:06:20.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:20.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:20.894 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/107897,v1:127.0.0.1:6811/107897] [v2:127.0.0.1:6812/107897,v1:127.0.0.1:6813/107897] exists,up 57d97048-098a-4a7a-9bf7-cd569bc6287f 2024-09-15T03:06:20.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:06:20.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:06:20.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:06:20.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: test_get_op_scheduler: get_op_scheduler 1 2024-09-15T03:06:20.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: get_op_scheduler: local id=1 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: get_op_scheduler: get_config osd 1 osd_op_queue 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:20.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.1.asok 2024-09-15T03:06:20.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:06:20.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-osd.1.asok config get osd_op_queue 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_get_op_scheduler: teardown td/ceph-helpers 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:06:20.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:06:20.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:06:20.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:06:21.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:06:21.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:06:21.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:06:21.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:06:21.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:06:21.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:06:21.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:06:21.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:06:21.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:06:21.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:06:21.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:06:21.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:06:21.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:06:21.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:06:21.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:06:21.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:21.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:21.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:06:21.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:06:21.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:06:21.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:06:21.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:06:21.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:06:21.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:06:21.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:06:21.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:06:21.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:06:21.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:06:21.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:06:21.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:06:21.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:06:21.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:06:21.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:06:21.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:06:21.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:06:21.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:21.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:21.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:06:21.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:06:21.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:06:21.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:06:21.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:06:21.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:21.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:21.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:06:21.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:06:21.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:06:21.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:06:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:06:21.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:06:21.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:06:21.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:06:21.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:06:21.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:21.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:21.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:06:21.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:06:21.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:06:21.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:06:21.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:06:21.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:06:21.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:06:21.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:06:21.717 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:06:21.717 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:06:21.717 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:06:21.718 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:06:21.718 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:21.718 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:21.718 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:06:21.718 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:06:21.718 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:06:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:06:21.794 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:21.794 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:21.794 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:06:21.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:06:21.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:06:21.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-15T03:06:21.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:06:21.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:06:21.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:06:21.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:06:21.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:06:21.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:06:22.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:06:22.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:06:22.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-15T03:06:22.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:06:22.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:06:22.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:06:22.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:06:22.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db456917-e91f-4b99-9e64-407a31487cca 2024-09-15T03:06:22.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 db456917-e91f-4b99-9e64-407a31487cca' 2024-09-15T03:06:22.735 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 db456917-e91f-4b99-9e64-407a31487cca 2024-09-15T03:06:22.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:06:22.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuT+ZmusxmLBAAb9/4SfmmP9jdc+ui94ygbw== 2024-09-15T03:06:22.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuT+ZmusxmLBAAb9/4SfmmP9jdc+ui94ygbw=="}' 2024-09-15T03:06:22.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db456917-e91f-4b99-9e64-407a31487cca -i td/ceph-helpers/0/new.json 2024-09-15T03:06:23.054 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:06:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:06:23.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuT+ZmusxmLBAAb9/4SfmmP9jdc+ui94ygbw== --osd-uuid db456917-e91f-4b99-9e64-407a31487cca 2024-09-15T03:06:23.089 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:23.087+0000 7f2cd515e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:23.100 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:23.099+0000 7f2cd515e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:23.101 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:23.100+0000 7f2cd515e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:23.102 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:23.100+0000 7f2cd515e440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:06:23.102 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:23.100+0000 7f2cd515e440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:06:23.102 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:23.100+0000 7f2cd515e440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:06:23.140 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:23.139+0000 7f2cd515e440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:06:28.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:06:28.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:06:28.475 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:06:28.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:06:28.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:06:28.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:06:28.958 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:06:28.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:06:28.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:06:28.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:06:28.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:06:28.980 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:28.978+0000 7f6f2ce15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:28.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:28.980+0000 7f6f2ce15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:28.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:28.981+0000 7f6f2ce15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:29.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:06:29.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:06:29.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:06:29.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:06:29.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:06:29.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:29.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:06:29.309 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:06:29.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:29.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:29.511 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:29.509+0000 7f6f2ce15440 -1 Falling back to public interface 2024-09-15T03:06:29.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:30.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:30.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:30.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:06:30.661 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:06:30.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:30.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:31.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:31.984 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:06:31.983+0000 7f6f2ce15440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:06:32.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:32.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:32.014 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:06:32.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:06:32.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:32.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:32.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:33.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:33.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:33.369 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:06:33.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:06:33.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:33.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:33.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:34.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:34.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:34.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:06:34.790 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:06:34.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:34.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:35.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:36.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:36.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:36.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:06:36.179 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:06:36.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:36.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:36.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:37.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:37.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:37.720 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:06:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:06:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:38.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:39.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:39.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:39.116 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:06:39.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:06:39.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:39.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:39.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:40.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:40.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:40.487 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:06:40.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:06:40.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:40.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:41.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:42.018 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:06:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:06:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:42.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:43.407 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:06:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:06:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:44.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:45.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:45.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:45.172 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:06:45.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:06:45.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:45.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:45.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:46.902 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:06:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:06:46.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:46.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:47.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:48.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:48.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:48.628 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:06:48.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:06:48.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:48.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:49.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:50.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:50.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:50.286 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:06:50.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:06:50.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:50.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:50.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:51.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:51.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:51.818 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:06:51.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:06:51.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:51.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:52.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:53.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:53.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:53.581 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:06:53.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:06:53.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:53.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:06:54.110 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/109930,v1:127.0.0.1:6803/109930] [v2:127.0.0.1:6804/109930,v1:127.0.0.1:6805/109930] exists,up db456917-e91f-4b99-9e64-407a31487cca 2024-09-15T03:06:54.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:06:54.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:06:54.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:06:54.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-09-15T03:06:54.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:06:54.581 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:06:54.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:06:54.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:06:55.151 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:06:55.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:06:56.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:06:57.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-09-15T03:06:57.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:06:57.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:06:57.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:06:57.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:06:57.310 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:06:57.310 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:06:57.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:06:57.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:06:57.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:06:57.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:06:57.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:06:57.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:06:57.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:06:57.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:06:57.407 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:06:57.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:06:57.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:06:57.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:06:57.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:06:57.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-15T03:06:57.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-15T03:06:57.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-15T03:06:57.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:06:57.870 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-15T03:06:57.870 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:06:57.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:06:57.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-15T03:06:57.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:06:57.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-15T03:06:57.873 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836487 2024-09-15T03:06:57.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-15T03:06:57.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:06:58.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-15T03:06:58.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:06:59.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:06:59.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:06:59.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-15T03:06:59.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:06:59.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:06:59.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:07:00.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:07:00.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:07:00.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:07:00.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:07:00.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:07:00.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:07:00.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:07:00.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:07:00.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:07:00.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:07:00.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:07:00.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:07:00.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:07:00.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:07:00.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:07:00.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-15T03:07:00.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-09-15T03:07:00.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-15T03:07:00.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-15T03:07:00.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-15T03:07:01.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:07:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:07:01.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:07:01.401 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:07:01.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:07:01.402 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:07:01.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:07:01.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:07:01.403 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:07:01.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:01.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:07:01.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:07:01.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:01.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:07:01.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:07:01.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:07:01.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:07:01.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:01.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:01.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:07:01.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:07:01.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_osds td/ceph-helpers 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:07:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:07:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:07:01.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:07:01.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:07:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:07:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:07:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:07:01.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:07:01.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:07:01.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:07:01.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:07:01.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:07:01.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:07:01.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:07:01.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:01.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:07:01.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:07:01.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:01.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:07:01.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:07:01.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:07:01.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:07:01.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:01.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:01.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:07:01.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:07:01.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:07:01.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:07:01.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:07:01.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:01.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:01.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:07:01.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:07:01.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:07:01.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:07:01.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:07:01.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-15T03:07:01.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:07:01.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:07:01.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:07:01.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:07:01.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:07:01.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-15T03:07:01.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:07:01.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:01.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:01.787 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:01.787 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:01.787 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:01.787 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:07:01.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T03:07:02.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:07:02.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:07:02.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:07:02.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:07:02.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:07:02.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:07:02.038 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:07:02.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:07:02.113 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:07:02.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:07:02.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-09-15T03:07:02.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:07:02.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:07:02.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:07:02.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:07:02.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:07:02.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:07:03.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:07:03.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:03.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:03.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:07:03.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:07:03.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:07:03.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:03.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:07:03.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:07:03.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:07:03.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:07:03.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a13404f2-d26f-4d1a-9cd2-e33dd00b938f 2024-09-15T03:07:03.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a13404f2-d26f-4d1a-9cd2-e33dd00b938f' 2024-09-15T03:07:03.066 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 a13404f2-d26f-4d1a-9cd2-e33dd00b938f 2024-09-15T03:07:03.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:07:03.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXT+ZmxvaTBBAApmZhQZCTNOfmtHFKallssA== 2024-09-15T03:07:03.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXT+ZmxvaTBBAApmZhQZCTNOfmtHFKallssA=="}' 2024-09-15T03:07:03.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a13404f2-d26f-4d1a-9cd2-e33dd00b938f -i td/ceph-helpers/0/new.json 2024-09-15T03:07:03.375 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:07:03.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:07:03.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXT+ZmxvaTBBAApmZhQZCTNOfmtHFKallssA== --osd-uuid a13404f2-d26f-4d1a-9cd2-e33dd00b938f 2024-09-15T03:07:03.446 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:03.409+0000 7fec9ed86440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:03.446 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:03.422+0000 7fec9ed86440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:03.446 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:03.423+0000 7fec9ed86440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:03.446 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:03.423+0000 7fec9ed86440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:07:03.447 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:03.423+0000 7fec9ed86440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:07:03.447 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:03.423+0000 7fec9ed86440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:07:03.462 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:03.462+0000 7fec9ed86440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:07:09.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:07:09.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:07:09.010 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:07:09.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:07:09.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:07:09.488 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:07:09.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:07:09.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:07:09.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:07:09.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:07:09.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:07:09.510 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:09.509+0000 7f570d935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:09.512 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:09.511+0000 7f570d935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:09.513 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:09.512+0000 7f570d935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:07:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:07:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:07:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:07:09.836 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:07:09.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:07:09.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:09.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:07:09.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:09.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:10.030 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:10.030+0000 7f570d935440 -1 Falling back to public interface 2024-09-15T03:07:10.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:11.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:11.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:11.185 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:07:11.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:07:11.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:11.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:11.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:12.505 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:12.504+0000 7f570d935440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:07:12.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:12.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:12.541 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:07:12.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:07:12.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:12.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:12.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:13.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:13.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:13.901 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:07:13.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:07:13.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:13.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:14.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:15.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:15.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:15.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:07:15.328 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:07:15.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:15.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:15.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:16.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:16.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:16.709 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:07:16.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:07:16.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:16.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:17.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:18.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:18.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:18.240 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:07:18.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:07:18.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:18.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:18.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:19.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:19.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:19.637 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:07:19.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:07:19.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:19.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:19.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:20.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:20.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:20.995 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:07:20.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:07:20.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:20.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:21.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:22.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:22.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:22.448 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:07:22.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:07:22.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:22.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:22.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:23.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:23.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:23.808 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:07:23.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:07:23.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:23.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:24.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:25.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:25.348 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:07:25.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:25.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:07:25.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:25.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:26.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:27.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:27.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:27.267 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:07:27.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:07:27.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:27.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:28.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:29.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:29.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:29.222 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:07:29.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:07:29.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:29.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:30.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:31.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:31.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:31.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:07:31.058 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:07:31.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:31.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:31.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:32.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:32.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:32.860 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:07:32.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:07:32.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:32.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:33.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:34.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:34.842 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:07:34.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:34.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:07:34.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:34.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:35.358 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/112246,v1:127.0.0.1:6803/112246] [v2:127.0.0.1:6804/112246,v1:127.0.0.1:6805/112246] exists,up a13404f2-d26f-4d1a-9cd2-e33dd00b938f 2024-09-15T03:07:35.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:07:35.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:07:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:07:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:07:35.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:07:35.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:07:35.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:07:35.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c299c14-d5ed-45d2-8176-c721312e4114 2024-09-15T03:07:35.366 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 6c299c14-d5ed-45d2-8176-c721312e4114 2024-09-15T03:07:35.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6c299c14-d5ed-45d2-8176-c721312e4114' 2024-09-15T03:07:35.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:07:35.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3T+ZmvmVtFhAANFJJREqgTFT1VzOJWlMzVg== 2024-09-15T03:07:35.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3T+ZmvmVtFhAANFJJREqgTFT1VzOJWlMzVg=="}' 2024-09-15T03:07:35.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c299c14-d5ed-45d2-8176-c721312e4114 -i td/ceph-helpers/1/new.json 2024-09-15T03:07:36.010 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:07:36.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:07:36.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3T+ZmvmVtFhAANFJJREqgTFT1VzOJWlMzVg== --osd-uuid 6c299c14-d5ed-45d2-8176-c721312e4114 2024-09-15T03:07:36.046 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:36.045+0000 7f2aa951d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:36.048 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:36.047+0000 7f2aa951d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:36.049 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:36.048+0000 7f2aa951d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:36.050 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:36.048+0000 7f2aa951d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:07:36.050 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:36.048+0000 7f2aa951d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:07:36.050 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:36.048+0000 7f2aa951d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:07:36.119 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:36.118+0000 7f2aa951d440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:07:42.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:07:42.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:07:42.043 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:07:42.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:07:42.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:07:42.532 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:07:42.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:07:42.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:07:42.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:07:42.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:07:42.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:07:42.555 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:42.553+0000 7f457723b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:42.557 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:42.555+0000 7f457723b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:42.560 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:42.556+0000 7f457723b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:42.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:07:42.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:07:42.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:07:42.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:07:42.884 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:07:42.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:07:42.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:42.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:07:42.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:42.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:43.074 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:43.073+0000 7f457723b440 -1 Falling back to public interface 2024-09-15T03:07:43.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:44.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:44.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:44.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:07:44.258 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:07:44.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:44.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:45.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:45.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:45.611 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:07:45.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:07:45.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:45.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:45.670 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:07:45.668+0000 7f457723b440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:07:46.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:47.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:47.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:47.013 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:07:47.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:07:47.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:47.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:47.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:48.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:48.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:48.429 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:07:48.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:07:48.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:48.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:48.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:49.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:49.786 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:07:49.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:49.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:07:49.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:49.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:50.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:51.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:51.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:51.153 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:07:51.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:07:51.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:51.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:51.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:52.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:52.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:52.534 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:07:52.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:07:52.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:52.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:52.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:53.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:53.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:53.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:07:53.948 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:07:53.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:53.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:54.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:55.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:55.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:55.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:07:55.497 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:07:55.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:55.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:55.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:56.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:56.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:56.925 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:07:56.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:07:56.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:56.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:57.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:58.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:58.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:58.363 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:07:58.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:07:58.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:58.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:07:59.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:00.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:00.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:00.193 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:08:00.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:08:00.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:00.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:00.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:01.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:01.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:01.963 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:08:01.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:08:01.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:01.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:02.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:03.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:03.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:03.684 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:08:03.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:08:03.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:03.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:04.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:05.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:05.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:05.459 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:08:05.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:08:05.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:05.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:06.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:07.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:07.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:07.278 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:08:07.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:08:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:08.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:09.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:09.178 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:08:09.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:08:09.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:09.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:09.578 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/113975,v1:127.0.0.1:6811/113975] [v2:127.0.0.1:6812/113975,v1:127.0.0.1:6813/113975] exists,up 6c299c14-d5ed-45d2-8176-c721312e4114 2024-09-15T03:08:09.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:08:09.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:08:09.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:08:09.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-09-15T03:08:09.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:08:10.046 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:08:10.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:08:10.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:08:10.922 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:08:10.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:08:11.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:08:13.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-09-15T03:08:13.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:08:13.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:08:13.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:08:13.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:08:13.591 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:08:13.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:08:13.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:08:13.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:08:13.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:08:13.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:08:13.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:08:13.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:08:13.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:08:13.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:08:13.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:08:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:08:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:1' 2024-09-15T03:08:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:08:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:08:14.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:08:14.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-15T03:08:14.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-15T03:08:14.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-15T03:08:14.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:08:14.141 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:08:14.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672967 2024-09-15T03:08:14.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672967 2024-09-15T03:08:14.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672967' 2024-09-15T03:08:14.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:08:14.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-15T03:08:14.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:08:14.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:08:14.266 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-15T03:08:14.266 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:08:14.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-15T03:08:14.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-15T03:08:14.268 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836492 2024-09-15T03:08:14.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:08:14.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-15T03:08:14.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:08:15.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:08:15.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:08:15.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-15T03:08:15.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:08:15.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672967 2024-09-15T03:08:15.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:08:15.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:08:15.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672967 2024-09-15T03:08:15.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:08:15.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672967 2024-09-15T03:08:15.976 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672967 2024-09-15T03:08:15.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-15T03:08:15.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:08:16.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-15T03:08:16.362 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:08:16.362 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:08:16.362 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:08:16.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:08:17.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:08:17.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:08:17.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:08:17.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:08:17.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:08:17.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:08:17.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:08:17.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-09-15T03:08:17.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:08:18.127 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:08:18.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:08:18.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:08:18.898 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:08:18.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:08:19.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:08:21.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-09-15T03:08:21.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-09-15T03:08:21.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-09-15T03:08:21.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-09-15T03:08:21.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-09-15T03:08:21.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:08:21.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-15T03:08:21.669 INFO:tasks.workunit.client.0.smithi137.stderr:0' 2024-09-15T03:08:21.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-15T03:08:21.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:08:21.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:08:21.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:08:21.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:08:21.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:08:21.777 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:08:21.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:08:21.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:08:21.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:08:21.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:08:21.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:08:21.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:08:21.779 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:08:21.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:08:21.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:08:21.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:08:21.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:08:21.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:21.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:21.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_pg td/ceph-helpers 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:08:21.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:08:21.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:08:21.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:08:21.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:08:21.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:08:21.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:08:21.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:08:21.795 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:08:21.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:08:21.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:08:21.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:08:21.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:08:21.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:08:21.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:08:21.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:08:21.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:08:21.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:08:21.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:08:21.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:08:21.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:21.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:21.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:08:21.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:08:21.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:08:21.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:08:21.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:08:21.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:21.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:21.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:08:21.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:08:21.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:08:21.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:08:21.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:08:22.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:08:22.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:08:22.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:08:22.173 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:08:22.173 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:22.173 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:22.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:08:22.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:08:22.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:08:22.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:08:22.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:08:22.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:08:22.491 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:08:22.492 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:22.492 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:22.492 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:08:22.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:08:22.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:08:22.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-09-15T03:08:22.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:08:22.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:08:22.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:08:22.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:08:22.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:08:22.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:08:23.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:08:23.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:08:23.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:08:23.410 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:08:23.410 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:23.411 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:23.411 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:08:23.411 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:08:23.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:08:23.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-09-15T03:08:23.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:08:23.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:08:23.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:08:23.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c5f3dd0-30fc-4a33-b1d4-e9a93544f517 2024-09-15T03:08:23.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3c5f3dd0-30fc-4a33-b1d4-e9a93544f517' 2024-09-15T03:08:23.442 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 3c5f3dd0-30fc-4a33-b1d4-e9a93544f517 2024-09-15T03:08:23.442 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:08:23.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnT+ZmGbXzGhAAyVEaGGGqpAGzn2KWTRv1LA== 2024-09-15T03:08:23.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnT+ZmGbXzGhAAyVEaGGGqpAGzn2KWTRv1LA=="}' 2024-09-15T03:08:23.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c5f3dd0-30fc-4a33-b1d4-e9a93544f517 -i td/ceph-helpers/0/new.json 2024-09-15T03:08:23.788 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:08:23.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:08:23.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnT+ZmGbXzGhAAyVEaGGGqpAGzn2KWTRv1LA== --osd-uuid 3c5f3dd0-30fc-4a33-b1d4-e9a93544f517 2024-09-15T03:08:23.822 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:23.820+0000 7fb4bd721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:23.841 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:23.840+0000 7fb4bd721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:23.843 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:23.841+0000 7fb4bd721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:23.843 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:23.841+0000 7fb4bd721440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:08:23.843 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:23.841+0000 7fb4bd721440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:08:23.843 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:23.841+0000 7fb4bd721440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:08:23.882 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:23.880+0000 7fb4bd721440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:08:29.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:08:29.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:08:29.246 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:08:29.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:08:29.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:08:29.730 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:08:29.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:08:29.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:08:29.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:08:29.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:08:29.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:08:29.752 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:29.751+0000 7fdfeea08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:29.754 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:29.753+0000 7fdfeea08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:29.755 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:29.754+0000 7fdfeea08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:08:30.079 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:08:30.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:30.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:30.268 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:30.267+0000 7fdfeea08440 -1 Falling back to public interface 2024-09-15T03:08:30.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:31.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:31.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:31.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:08:31.428 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:08:31.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:31.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:31.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:32.750 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:08:32.748+0000 7fdfeea08440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:08:32.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:32.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:32.786 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:08:32.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:08:32.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:32.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:33.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:34.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:34.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:34.319 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:08:34.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:08:34.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:34.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:34.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:35.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:35.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:35.686 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:08:35.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:08:35.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:35.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:36.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:37.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:37.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:37.078 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:08:37.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:08:37.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:37.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:37.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:38.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:38.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:38.567 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:08:38.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:08:38.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:38.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:38.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:39.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:39.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:39.944 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:08:39.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:08:39.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:39.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:40.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:41.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:41.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:41.447 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:08:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:08:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:41.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:42.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:42.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:42.852 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:08:42.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:08:42.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:42.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:43.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:44.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:44.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:44.256 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:08:44.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:08:44.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:44.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:44.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:45.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:45.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:45.947 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:08:45.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:08:45.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:45.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:46.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:47.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:47.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:47.667 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:08:47.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:08:47.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:47.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:48.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:49.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:49.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:49.421 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:08:49.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:08:49.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:49.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:50.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:51.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:51.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:51.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:08:51.134 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:08:51.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:51.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:51.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:52.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:52.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:52.937 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:08:52.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:08:52.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:52.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:53.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:54.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:54.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:54.430 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:08:54.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:08:54.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:54.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:54.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:55.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:55.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:55.810 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:08:55.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:08:55.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:55.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:56.178 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/116437,v1:127.0.0.1:6803/116437] [v2:127.0.0.1:6804/116437,v1:127.0.0.1:6805/116437] exists,up 3c5f3dd0-30fc-4a33-b1d4-e9a93544f517 2024-09-15T03:08:56.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:08:56.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:08:56.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:08:56.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-09-15T03:08:56.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:08:56.530 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:08:56.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:08:56.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:08:57.037 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:08:57.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:08:58.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:08:59.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-09-15T03:08:59.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:08:59.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:08:59.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:08:59.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:08:59.087 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:08:59.087 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:08:59.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:08:59.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:08:59.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:08:59.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:08:59.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:08:59.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:08:59.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:08:59.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:08:59.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:08:59.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:08:59.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:08:59.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:08:59.536 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:08:59.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-15T03:08:59.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-15T03:08:59.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-15T03:08:59.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:08:59.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-15T03:08:59.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:08:59.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:08:59.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-15T03:08:59.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:08:59.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-15T03:08:59.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-15T03:08:59.644 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836486 2024-09-15T03:08:59.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:08:59.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-15T03:08:59.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:09:00.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:09:00.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:09:01.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-15T03:09:01.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:09:01.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:09:01.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:09:01.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:09:01.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:09:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:09:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:09:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:09:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:09:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:09:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:09:02.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:09:02.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:09:02.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:09:02.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:09:02.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:09:02.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:09:02.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:09:02.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-09-15T03:09:02.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-15T03:09:02.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-09-15T03:09:02.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-09-15T03:09:02.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-09-15T03:09:02.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T03:09:03.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-09-15T03:09:03.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:09:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:09:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:09:03.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:09:03.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:09:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:09:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:09:03.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:09:03.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:09:03.139 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:09:03.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:09:03.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:09:03.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:09:03.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:09:03.141 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:09:03.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:03.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:09:03.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:09:03.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:03.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:09:03.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:09:03.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:09:03.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:09:03.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:03.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:03.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:09:03.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:09:03.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:09:03.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:09:03.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_primary td/ceph-helpers 2024-09-15T03:09:03.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-09-15T03:09:03.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-09-15T03:09:03.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:09:03.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:09:03.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:09:03.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:09:03.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:09:03.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:09:03.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:09:03.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:09:03.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:09:03.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:03.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:09:03.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:09:03.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:03.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:09:03.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:09:03.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:09:03.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:09:03.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:03.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:03.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:09:03.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:09:03.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:09:03.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:09:03.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:09:03.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:03.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:03.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:09:03.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:09:03.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:09:03.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:09:03.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:09:03.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:09:03.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:03.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:03.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:03.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:09:03.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:09:03.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:09:03.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:09:03.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:09:03.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:09:03.773 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:09:03.773 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:09:03.774 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:03.849 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:09:03.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:09:03.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:09:03.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-09-15T03:09:03.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-09-15T03:09:03.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:09:03.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:09:03.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:09:03.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:09:03.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:09:03.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:09:04.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:09:04.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:09:04.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:09:04.758 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:09:04.758 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:04.758 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:04.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:04.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:09:04.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:09:04.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-09-15T03:09:04.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:09:04.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:09:04.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:09:04.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:09:04.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:09:04.789 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:09:04.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba5dd10e-7f34-4e1c-b19f-24328d6caf1c 2024-09-15T03:09:04.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ba5dd10e-7f34-4e1c-b19f-24328d6caf1c' 2024-09-15T03:09:04.790 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 ba5dd10e-7f34-4e1c-b19f-24328d6caf1c 2024-09-15T03:09:04.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:09:04.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQT+ZmNIm4LxAAdRE1Zk2E92SIY6SMWeWlug== 2024-09-15T03:09:04.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQT+ZmNIm4LxAAdRE1Zk2E92SIY6SMWeWlug=="}' 2024-09-15T03:09:04.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba5dd10e-7f34-4e1c-b19f-24328d6caf1c -i td/ceph-helpers/0/new.json 2024-09-15T03:09:05.112 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:09:05.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:09:05.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQT+ZmNIm4LxAAdRE1Zk2E92SIY6SMWeWlug== --osd-uuid ba5dd10e-7f34-4e1c-b19f-24328d6caf1c 2024-09-15T03:09:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:05.146+0000 7fce94b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:05.173 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:05.173+0000 7fce94b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:05.174 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:05.174+0000 7fce94b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:05.175 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:05.174+0000 7fce94b21440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:09:05.175 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:05.174+0000 7fce94b21440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:09:05.175 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:05.174+0000 7fce94b21440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:09:05.213 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:05.213+0000 7fce94b21440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:09:10.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:09:10.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:09:10.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:09:10.650 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:09:10.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:09:11.131 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:09:11.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:09:11.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:09:11.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:09:11.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:09:11.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:09:11.153 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:11.153+0000 7f3733957440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:11.155 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:11.155+0000 7f3733957440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:11.157 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:11.156+0000 7f3733957440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:11.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:09:11.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:09:11.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:09:11.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:09:11.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:09:11.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:11.483 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:09:11.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:09:11.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:11.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:11.686 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:11.685+0000 7f3733957440 -1 Falling back to public interface 2024-09-15T03:09:11.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:12.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:12.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:12.854 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:09:12.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:09:12.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:12.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:13.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:14.183 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:14.181+0000 7f3733957440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:09:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:14.212 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:09:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:09:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:14.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:15.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:15.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:15.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:09:15.786 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:09:15.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:15.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:16.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:17.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:17.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:17.189 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:09:17.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:09:17.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:17.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:17.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:18.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:18.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:18.547 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:09:18.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:09:18.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:18.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:19.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:20.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:20.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:20.041 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:09:20.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:09:20.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:20.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:20.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:21.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:21.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:21.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:09:21.397 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:09:21.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:21.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:21.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:22.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:22.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:22.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:09:22.871 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:09:22.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:22.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:24.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:24.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:24.234 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:09:24.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:09:24.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:24.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:24.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:25.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:25.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:25.640 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:09:25.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:09:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:25.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:26.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:27.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:27.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:27.319 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:09:27.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:09:27.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:27.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:28.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:29.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:29.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:29.091 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:09:29.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:09:29.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:29.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:30.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:30.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:30.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:09:30.807 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:09:30.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:30.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:31.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:32.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:32.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:32.538 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:09:32.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:09:32.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:32.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:33.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:34.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:34.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:34.420 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:09:34.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:09:34.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:34.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:35.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:35.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:35.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:09:35.889 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:09:35.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:35.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:36.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:37.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:37.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:37.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:09:37.562 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:09:37.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:37.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:37.938 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/118777,v1:127.0.0.1:6803/118777] [v2:127.0.0.1:6804/118777,v1:127.0.0.1:6805/118777] exists,up ba5dd10e-7f34-4e1c-b19f-24328d6caf1c 2024-09-15T03:09:37.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:09:37.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:09:37.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:09:37.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-09-15T03:09:37.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:09:38.279 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:09:38.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:09:38.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:09:38.812 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:09:38.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:09:39.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:09:40.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-09-15T03:09:40.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:09:40.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:09:40.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:09:40.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:09:40.963 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:40.963 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:40.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:40.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:40.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:41.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:09:41.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:09:41.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:09:41.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:09:41.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:09:41.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:09:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:09:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:09:41.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:09:41.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:09:41.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-15T03:09:41.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-15T03:09:41.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-15T03:09:41.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:09:41.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-15T03:09:41.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:09:41.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:09:41.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-15T03:09:41.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:09:41.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-15T03:09:41.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-15T03:09:41.515 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836491 2024-09-15T03:09:41.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:09:41.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-15T03:09:41.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:09:42.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:09:42.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:09:43.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-15T03:09:43.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:09:43.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:09:43.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:09:43.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:09:43.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:09:43.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:09:43.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:09:43.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:09:43.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:09:43.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:09:43.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:09:44.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:09:44.056 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:09:44.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:09:44.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:09:44.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:09:44.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:09:44.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-09-15T03:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-15T03:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-15T03:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-15T03:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T03:09:44.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-09-15T03:09:44.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-09-15T03:09:44.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:09:44.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:09:44.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:09:44.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:09:44.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:09:44.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:09:44.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:09:44.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:09:45.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:09:45.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:09:45.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:09:45.054 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:09:45.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:09:45.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:09:45.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:09:45.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:45.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:09:45.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:09:45.056 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:45.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:09:45.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:09:45.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:09:45.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:09:45.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:45.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:45.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:09:45.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:09:45.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:09:45.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:09:45.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_timeout_delays td/ceph-helpers 2024-09-15T03:09:45.065 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-09-15T03:09:45.066 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.066 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-09-15T03:09:45.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-09-15T03:09:45.077 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.077 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-09-15T03:09:45.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-09-15T03:09:45.096 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.096 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-09-15T03:09:45.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-09-15T03:09:45.115 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.115 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-09-15T03:09:45.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-09-15T03:09:45.140 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.140 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-09-15T03:09:45.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-09-15T03:09:45.165 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.165 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-09-15T03:09:45.190 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-09-15T03:09:45.191 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.191 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-09-15T03:09:45.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-09-15T03:09:45.221 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.222 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-09-15T03:09:45.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-09-15T03:09:45.261 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.261 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-09-15T03:09:45.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-09-15T03:09:45.300 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.300 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-09-15T03:09:45.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-09-15T03:09:45.339 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.339 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-09-15T03:09:45.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-09-15T03:09:45.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-09-15T03:09:45.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-09-15T03:09:45.470 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:09:45.470 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:09:45.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:09:45.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:09:45.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:09:45.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-09-15T03:09:45.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:09:45.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_is_clean td/ceph-helpers 2024-09-15T03:09:45.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:09:45.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:09:45.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:09:45.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:09:45.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:09:45.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:09:45.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:09:45.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:09:45.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:09:45.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:09:45.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:45.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:09:45.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:09:45.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:09:45.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:09:45.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:09:45.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:09:45.715 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:09:45.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:45.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:45.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:09:45.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:09:45.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:09:45.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:09:45.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:09:45.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:45.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:45.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:09:45.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:09:46.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:09:46.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:09:46.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:09:46.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:09:46.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:46.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:46.075 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:46.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:09:46.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:09:46.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:09:46.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:09:46.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:09:46.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:09:46.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:09:46.293 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:09:46.293 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:09:46.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:09:46.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:09:46.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:46.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:46.295 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:09:46.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:09:46.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:09:46.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:46.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:09:46.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:09:46.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:09:46.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-09-15T03:09:46.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:09:46.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:09:46.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:09:46.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:09:46.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:09:46.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:09:47.278 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:09:47.278 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:09:47.278 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:09:47.278 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:09:47.278 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:47.278 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:47.278 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:47.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:09:47.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:09:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:09:47.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:09:47.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:09:47.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:09:47.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:09:47.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f851ebd0-b999-48da-9a9b-876ffdda05fb 2024-09-15T03:09:47.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f851ebd0-b999-48da-9a9b-876ffdda05fb' 2024-09-15T03:09:47.309 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 f851ebd0-b999-48da-9a9b-876ffdda05fb 2024-09-15T03:09:47.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:09:47.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7T+ZmHcgKExAA7J7Bz2cquS35RzDBwYlK+w== 2024-09-15T03:09:47.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7T+ZmHcgKExAA7J7Bz2cquS35RzDBwYlK+w=="}' 2024-09-15T03:09:47.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f851ebd0-b999-48da-9a9b-876ffdda05fb -i td/ceph-helpers/0/new.json 2024-09-15T03:09:47.656 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:09:47.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:09:47.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7T+ZmHcgKExAA7J7Bz2cquS35RzDBwYlK+w== --osd-uuid f851ebd0-b999-48da-9a9b-876ffdda05fb 2024-09-15T03:09:47.691 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:47.691+0000 7f8121889440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:47.701 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:47.701+0000 7f8121889440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:47.702 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:47.702+0000 7f8121889440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:47.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:47.702+0000 7f8121889440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:09:47.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:47.702+0000 7f8121889440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:09:47.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:47.702+0000 7f8121889440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:09:47.742 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:47.741+0000 7f8121889440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:09:53.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:09:53.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:09:53.391 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:09:53.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:09:53.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:09:53.869 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:09:53.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:09:53.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:09:53.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:09:53.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:09:53.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:09:53.892 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:53.891+0000 7f8dc300e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:53.894 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:53.893+0000 7f8dc300e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:53.895 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:53.894+0000 7f8dc300e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:54.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:09:54.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:09:54.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:09:54.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:09:54.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:09:54.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:54.219 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:09:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:09:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:54.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:54.413 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:54.413+0000 7f8dc300e440 -1 Falling back to public interface 2024-09-15T03:09:54.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:55.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:55.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:55.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:09:55.574 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:09:55.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:55.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:55.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:56.913 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:09:56.912+0000 7f8dc300e440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:09:56.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:56.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:56.924 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:09:56.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:09:56.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:56.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:57.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:58.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:58.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:58.288 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:09:58.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:09:58.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:58.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:09:58.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:59.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:59.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:59.734 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:09:59.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:09:59.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:59.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:00.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:01.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:01.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:01.131 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:10:01.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:10:01.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:01.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:01.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:02.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:02.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:02.597 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:10:02.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:10:02.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:02.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:02.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:03.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:03.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:03.959 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:10:03.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:10:03.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:03.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:04.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:05.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:05.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:05.316 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:10:05.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:10:05.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:05.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:05.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:06.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:06.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:06.720 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:10:06.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:10:06.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:06.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:07.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:08.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:08.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:08.109 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:10:08.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:10:08.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:08.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:08.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:09.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:09.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:09.760 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:10:09.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:10:09.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:09.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:10.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:11.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:11.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:11.206 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:10:11.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:10:11.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:11.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:12.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:13.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:13.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:13.279 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:10:13.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:10:13.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:13.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:13.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:14.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:14.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:14.720 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:10:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:10:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:14.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:15.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:16.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:16.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:16.298 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:10:16.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:10:16.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:16.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:17.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:18.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:18.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:18.010 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:10:18.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:10:18.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:18.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:19.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:20.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:20.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:20.086 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:10:20.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:10:20.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:20.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:20.485 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/121522,v1:127.0.0.1:6803/121522] [v2:127.0.0.1:6804/121522,v1:127.0.0.1:6805/121522] exists,up f851ebd0-b999-48da-9a9b-876ffdda05fb 2024-09-15T03:10:20.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:10:20.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:10:20.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:10:20.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-09-15T03:10:20.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:10:20.832 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:10:20.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:10:20.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:10:21.338 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:10:21.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:10:22.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:10:23.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-09-15T03:10:23.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:10:23.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:10:23.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:10:23.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:10:23.366 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:10:23.366 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:10:23.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:10:23.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:10:23.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:10:23.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:10:23.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:10:23.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:10:23.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:10:23.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:10:23.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:10:23.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:10:23.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:10:23.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:10:23.812 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:10:23.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836490 2024-09-15T03:10:23.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836490 2024-09-15T03:10:23.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490' 2024-09-15T03:10:23.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:10:23.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836490 2024-09-15T03:10:23.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:10:23.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:10:23.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836490 2024-09-15T03:10:23.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:10:23.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836490 2024-09-15T03:10:23.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-15T03:10:23.921 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836490 2024-09-15T03:10:23.922 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:10:24.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-15T03:10:24.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:10:25.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:10:25.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:10:25.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-15T03:10:25.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:10:25.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:10:25.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:10:26.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:10:26.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:10:26.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:10:26.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:10:26.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:10:26.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:10:26.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:10:26.130 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:10:26.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:10:26.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:10:26.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:10:26.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:10:26.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:10:26.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:10:26.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:10:26.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-09-15T03:10:26.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-09-15T03:10:26.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:10:26.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:10:27.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-09-15T03:10:27.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-09-15T03:10:27.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-09-15T03:10:27.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:10:27.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:10:27.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:10:27.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:10:27.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:10:27.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-09-15T03:10:27.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-09-15T03:10:27.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:10:27.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:10:27.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:10:27.787 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:10:27.787 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:10:27.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:10:27.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:10:27.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:10:27.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:10:27.892 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:10:27.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:10:27.893 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:10:27.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:10:27.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:10:27.894 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:10:27.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:27.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:10:27.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:10:27.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:27.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:10:27.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:10:27.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:10:27.903 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:10:27.903 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:27.903 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:27.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:10:27.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:10:27.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:10:27.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_kill_daemon td/ceph-helpers 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:10:27.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:10:27.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:10:27.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:10:27.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:10:27.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:10:27.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:10:27.908 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:10:27.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:10:27.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:10:27.908 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:10:27.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:27.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:10:27.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:10:27.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:27.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:10:27.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:10:27.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:10:27.912 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:10:27.912 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:27.912 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:27.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:10:27.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:10:27.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:10:27.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:10:27.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:10:27.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:27.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:27.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:10:27.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:10:27.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:10:28.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:10:28.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:10:28.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:10:28.284 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:10:28.284 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:28.285 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:28.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:10:28.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:10:28.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:10:28.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:10:28.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:10:28.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:10:28.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:10:28.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:10:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:28.632 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:28.632 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:28.632 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:10:28.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:10:28.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:10:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-09-15T03:10:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:10:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:10:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:10:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:10:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:10:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:10:29.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:10:29.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:10:29.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:10:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:10:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:29.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:10:29.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:10:29.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:10:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:10:29.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:10:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:10:29.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:10:29.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:10:29.577 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:10:29.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=227f0835-46cb-417b-a7b2-4cf1cd286664 2024-09-15T03:10:29.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 227f0835-46cb-417b-a7b2-4cf1cd286664' 2024-09-15T03:10:29.577 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 227f0835-46cb-417b-a7b2-4cf1cd286664 2024-09-15T03:10:29.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:10:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlUOZmTvMFIxAAYu00WVUJY/MPs37YTyt/jQ== 2024-09-15T03:10:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlUOZmTvMFIxAAYu00WVUJY/MPs37YTyt/jQ=="}' 2024-09-15T03:10:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 227f0835-46cb-417b-a7b2-4cf1cd286664 -i td/ceph-helpers/0/new.json 2024-09-15T03:10:29.873 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:10:29.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:10:29.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlUOZmTvMFIxAAYu00WVUJY/MPs37YTyt/jQ== --osd-uuid 227f0835-46cb-417b-a7b2-4cf1cd286664 2024-09-15T03:10:29.907 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:29.906+0000 7f053008b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:29.927 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:29.926+0000 7f053008b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:29.928 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:29.927+0000 7f053008b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:29.929 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:29.927+0000 7f053008b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:10:29.929 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:29.927+0000 7f053008b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:10:29.929 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:29.927+0000 7f053008b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:10:29.959 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:29.958+0000 7f053008b440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:10:35.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:10:35.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:10:35.514 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:10:35.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:10:35.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:10:36.003 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:10:36.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:10:36.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:10:36.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:10:36.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:10:36.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:10:36.026 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:36.024+0000 7ff368d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:36.028 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:36.026+0000 7ff368d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:36.029 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:36.027+0000 7ff368d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:10:36.357 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:10:36.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:36.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:36.545 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:36.544+0000 7ff368d21440 -1 Falling back to public interface 2024-09-15T03:10:36.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:37.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:37.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:37.708 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:10:37.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:10:37.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:37.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:38.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:39.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:39.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:39.067 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:10:39.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:10:39.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:39.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:39.069 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:10:39.068+0000 7ff368d21440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:10:39.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:40.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:40.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:40.453 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:10:40.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:10:40.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:40.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:40.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:41.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:41.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:41.958 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:10:41.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:10:41.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:41.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:42.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:43.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:43.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:43.320 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:10:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:10:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:43.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:44.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:44.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:44.780 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:10:44.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:10:44.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:44.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:45.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:46.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:46.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:46.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:10:46.141 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:10:46.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:46.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:46.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:47.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:47.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:47.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:10:47.530 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:10:47.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:47.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:47.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:48.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:48.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:48.919 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:10:48.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:10:48.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:48.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:49.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:50.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:50.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:50.349 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:10:50.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:10:50.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:50.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:50.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:51.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:51.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:51.717 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:10:51.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:10:51.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:51.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:52.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:53.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:53.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:53.193 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:10:53.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:10:53.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:53.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:53.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:54.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:54.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:54.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:10:54.833 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:10:54.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:54.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:55.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:56.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:56.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:56.358 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:10:56.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:10:56.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:56.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:57.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:58.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:58.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:58.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:10:58.102 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:10:58.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:58.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:58.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:59.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:59.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:59.920 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:10:59.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:10:59.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:59.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:00.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:01.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:01.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:01.937 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:11:01.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:11:01.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:01.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:02.332 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/123887,v1:127.0.0.1:6803/123887] [v2:127.0.0.1:6804/123887,v1:127.0.0.1:6805/123887] exists,up 227f0835-46cb-417b-a7b2-4cf1cd286664 2024-09-15T03:11:02.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:11:02.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:11:02.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:11:02.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-09-15T03:11:02.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-09-15T03:11:02.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-09-15T03:11:02.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-15T03:11:02.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-09-15T03:11:02.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-15T03:11:02.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=123887 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123887 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-15T03:11:02.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-09-15T03:11:03.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123887 2024-09-15T03:11:03.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-15T03:11:03.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-15T03:11:03.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-09-15T03:11:03.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-09-15T03:11:03.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-09-15T03:11:03.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=123887 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 123887 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-15T03:11:03.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-15T03:11:03.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123887 2024-09-15T03:11:03.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-15T03:11:03.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-15T03:11:03.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-09-15T03:11:03.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123887 2024-09-15T03:11:03.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-15T03:11:03.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-15T03:11:03.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-15T03:11:03.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-09-15T03:11:03.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-09-15T03:11:03.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-09-15T03:11:03.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-15T03:11:03.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-09-15T03:11:03.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-09-15T03:11:03.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=122934 2024-09-15T03:11:03.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-15T03:11:03.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-15T03:11:03.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-15T03:11:03.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 122934 2024-09-15T03:11:03.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-15T03:11:03.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-15T03:11:03.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-15T03:11:03.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 122934 2024-09-15T03:11:03.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-15T03:11:03.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-15T03:11:03.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-15T03:11:03.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-09-15T03:11:03.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-09-15T03:11:03.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-09-15T03:11:03.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-15T03:11:03.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-09-15T03:11:03.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-09-15T03:11:03.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=122849 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 122849 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-15T03:11:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-15T03:11:03.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-15T03:11:03.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 122849 2024-09-15T03:11:03.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-15T03:11:03.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-15T03:11:03.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-15T03:11:03.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-09-15T03:11:08.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-09-15T03:11:08.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:11:08.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:11:08.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:11:08.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:08.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:08.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:08.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:08.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:08.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:11:08.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:11:08.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:11:08.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:11:08.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:11:08.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:11:08.864 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:11:08.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:08.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:11:08.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:11:08.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:08.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:11:08.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:11:08.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:11:08.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:11:08.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:08.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:08.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:11:08.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:11:08.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:11:08.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:11:08.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_kill_daemons td/ceph-helpers 2024-09-15T03:11:08.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-09-15T03:11:08.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:08.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:08.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:11:08.878 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:11:08.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:11:08.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:11:08.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:11:08.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:11:08.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:11:08.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:08.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:11:08.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:11:08.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:08.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:11:08.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:11:08.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:11:08.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:11:08.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:08.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:08.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:11:08.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:11:08.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:11:08.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:11:08.888 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:11:08.888 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:08.888 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:08.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:11:08.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:11:08.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:11:08.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:11:08.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:11:09.229 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:11:09.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:11:09.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:11:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:11:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:09.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:11:09.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:11:09.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:11:09.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:11:09.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:11:09.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:11:09.483 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:11:09.483 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:11:09.483 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:11:09.484 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:11:09.484 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:11:09.484 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:09.484 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:09.484 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:11:09.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:11:09.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:11:09.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:11:09.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:11:09.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:11:09.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-09-15T03:11:09.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:11:09.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:11:09.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:11:09.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:11:09.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:11:09.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:11:10.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:11:10.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:11:10.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:11:10.504 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:11:10.504 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:10.504 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:10.504 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:10.504 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:11:10.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:11:10.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-09-15T03:11:10.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:11:10.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:11:10.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:11:10.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:11:10.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:11:10.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:11:10.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:11:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:11:10.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6165903a-3378-47f7-ac61-ff2a127b0381 2024-09-15T03:11:10.535 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 6165903a-3378-47f7-ac61-ff2a127b0381 2024-09-15T03:11:10.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6165903a-3378-47f7-ac61-ff2a127b0381' 2024-09-15T03:11:10.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:11:10.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOUOZm/umDIBAAcf7K8OYjrTiZB0Gnr6Furw== 2024-09-15T03:11:10.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOUOZm/umDIBAAcf7K8OYjrTiZB0Gnr6Furw=="}' 2024-09-15T03:11:10.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6165903a-3378-47f7-ac61-ff2a127b0381 -i td/ceph-helpers/0/new.json 2024-09-15T03:11:10.852 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:11:10.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:11:10.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOUOZm/umDIBAAcf7K8OYjrTiZB0Gnr6Furw== --osd-uuid 6165903a-3378-47f7-ac61-ff2a127b0381 2024-09-15T03:11:10.886 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:10.885+0000 7f1c3624f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:10.898 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:10.896+0000 7f1c3624f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:10.899 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:10.897+0000 7f1c3624f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:10.899 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:10.898+0000 7f1c3624f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:11:10.899 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:10.898+0000 7f1c3624f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:11:10.900 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:10.898+0000 7f1c3624f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:11:10.938 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:10.937+0000 7f1c3624f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:11:16.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:11:16.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:11:16.232 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:11:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:11:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:11:16.716 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:11:16.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:11:16.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:11:16.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:11:16.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:11:16.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:11:16.738 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:16.737+0000 7f8da2351440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:16.741 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:16.739+0000 7f8da2351440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:16.742 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:16.740+0000 7f8da2351440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:17.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:11:17.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:11:17.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:11:17.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:11:17.069 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:11:17.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:11:17.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:17.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:11:17.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:17.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:17.261 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:17.260+0000 7f8da2351440 -1 Falling back to public interface 2024-09-15T03:11:17.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:18.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:18.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:18.427 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:11:18.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:11:18.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:18.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:18.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:19.768 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:19.766+0000 7f8da2351440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:11:19.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:19.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:19.786 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:11:19.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:11:19.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:19.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:20.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:21.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:21.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:21.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:11:21.153 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:11:21.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:21.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:21.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:22.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:22.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:22.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:11:22.630 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:11:22.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:22.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:23.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:24.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:24.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:24.011 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:11:24.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:11:24.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:24.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:24.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:25.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:25.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:25.476 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:11:25.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:11:25.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:25.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:25.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:26.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:26.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:26.883 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:11:26.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:11:26.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:26.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:27.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:28.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:28.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:28.268 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:11:28.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:11:28.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:28.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:28.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:29.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:29.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:29.632 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:11:29.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:11:29.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:29.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:30.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:31.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:31.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:31.037 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:11:31.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:11:31.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:31.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:31.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:32.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:32.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:32.484 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:11:32.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:11:32.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:32.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:32.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:33.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:33.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:33.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:11:33.949 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:11:33.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:33.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:34.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:35.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:35.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:35.639 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:11:35.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:11:35.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:35.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:36.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:37.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:37.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:37.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:11:37.209 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:11:37.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:37.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:37.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:38.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:38.903 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:11:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:11:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:38.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:39.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:40.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:40.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:40.685 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:11:40.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:11:40.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:40.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:41.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:42.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:42.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:42.743 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:11:42.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:11:42.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:42.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:43.184 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/125932,v1:127.0.0.1:6803/125932] [v2:127.0.0.1:6804/125932,v1:127.0.0.1:6805/125932] exists,up 6165903a-3378-47f7-ac61-ff2a127b0381 2024-09-15T03:11:43.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:11:43.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:11:43.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:11:43.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-09-15T03:11:43.186 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:43.186 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:43.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:43.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:43.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:44.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-09-15T03:11:44.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-09-15T03:11:44.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:44.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:44.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:44.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:44.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:44.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:11:44.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-09-15T03:11:44.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:44.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:44.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:44.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:44.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:44.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:11:44.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-09-15T03:11:44.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:44.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:44.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:44.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:44.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:44.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:11:44.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-09-15T03:11:49.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-09-15T03:11:49.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:11:49.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:11:49.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:11:49.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:49.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:49.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:49.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:49.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:49.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:11:49.924 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:11:49.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:11:49.925 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:11:49.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:11:49.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:11:49.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:11:49.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:49.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:11:49.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:11:49.928 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:49.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:11:49.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:11:49.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:11:49.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:11:49.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:49.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:49.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_objectstore_tool td/ceph-helpers 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:11:49.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:11:49.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:11:49.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:11:49.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:11:49.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:11:49.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:11:49.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:11:49.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:11:49.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:11:49.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:11:49.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:11:49.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:11:49.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:11:49.943 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:11:49.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:49.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:11:49.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:11:49.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:11:49.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:11:49.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:11:49.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:11:49.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:11:49.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:49.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:49.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:11:49.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:11:49.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:11:49.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:11:49.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:11:49.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:49.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:49.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:11:49.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:11:49.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:11:49.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:11:50.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:11:50.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:11:50.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:11:50.313 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:11:50.313 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:50.313 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:50.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:50.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:11:50.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:11:50.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:11:50.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:11:50.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:11:50.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:11:50.595 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:11:50.595 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:11:50.595 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:11:50.595 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:11:50.595 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:11:50.595 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:50.596 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:50.596 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:11:50.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:11:50.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:11:50.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:11:50.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:11:50.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:11:50.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-09-15T03:11:50.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-09-15T03:11:50.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:11:50.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:11:50.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:11:50.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:11:50.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:11:50.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:11:51.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:11:51.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:11:51.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:11:51.600 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:11:51.600 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:51.600 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:51.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:51.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:11:51.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:11:51.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:11:51.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:51.627 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:11:51.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:11:51.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:11:51.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:11:51.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:11:51.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd81369b-cbf0-4563-8f75-2b6575d4c8ae 2024-09-15T03:11:51.629 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 bd81369b-cbf0-4563-8f75-2b6575d4c8ae 2024-09-15T03:11:51.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bd81369b-cbf0-4563-8f75-2b6575d4c8ae' 2024-09-15T03:11:51.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:11:51.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3UOZmT6QpJhAAc0gWetvXK1lyfarp7noOEA== 2024-09-15T03:11:51.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3UOZmT6QpJhAAc0gWetvXK1lyfarp7noOEA=="}' 2024-09-15T03:11:51.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd81369b-cbf0-4563-8f75-2b6575d4c8ae -i td/ceph-helpers/0/new.json 2024-09-15T03:11:51.942 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:11:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:11:51.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3UOZmT6QpJhAAc0gWetvXK1lyfarp7noOEA== --osd-uuid bd81369b-cbf0-4563-8f75-2b6575d4c8ae 2024-09-15T03:11:51.976 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:51.976+0000 7fe4a631f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:51.988 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:51.988+0000 7fe4a631f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:51.989 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:51.989+0000 7fe4a631f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:51.990 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:51.989+0000 7fe4a631f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:11:51.990 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:51.989+0000 7fe4a631f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:11:51.990 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:51.989+0000 7fe4a631f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:11:52.029 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:52.028+0000 7fe4a631f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:11:57.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:11:57.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:11:57.717 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:11:57.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:11:57.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:11:58.198 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:11:58.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:11:58.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:11:58.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:11:58.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:11:58.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:11:58.220 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:58.220+0000 7fd430d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:58.222 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:58.222+0000 7fd430d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:58.223 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:58.223+0000 7fd430d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:58.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:11:58.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:11:58.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:11:58.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:11:58.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:11:58.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:58.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:11:58.548 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:11:58.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:58.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:11:58.739 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:11:58.739+0000 7fd430d22440 -1 Falling back to public interface 2024-09-15T03:11:58.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:59.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:59.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:59.918 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:11:59.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:11:59.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:59.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:00.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:01.222 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:01.220+0000 7fd430d22440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:12:01.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:01.277 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:12:01.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:12:01.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:01.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:01.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:02.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:02.713 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:12:02.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:02.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:12:02.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:02.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:03.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:04.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:04.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:04.190 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:12:04.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:12:04.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:04.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:04.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:05.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:05.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:05.573 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:12:05.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:12:05.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:05.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:06.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:07.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:07.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:07.073 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:12:07.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:12:07.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:07.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:07.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:08.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:08.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:08.437 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:12:08.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:12:08.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:08.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:08.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:09.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:09.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:09.977 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:12:09.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:12:09.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:09.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:10.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:11.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:11.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:11.349 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:12:11.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:12:11.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:11.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:11.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:12.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:12.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:12.720 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:12:12.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:12:12.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:12.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:13.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:14.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:14.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:14.327 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:12:14.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:12:14.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:14.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:14.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:15.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:15.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:15.942 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:12:15.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:12:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:16.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:17.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:17.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:17.648 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:12:17.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:12:17.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:17.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:18.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:19.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:19.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:19.886 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:12:19.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:12:19.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:19.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:20.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:21.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:21.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:21.663 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:12:21.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:12:21.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:21.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:22.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:23.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:23.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:23.064 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:12:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:12:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:23.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:24.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:24.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:24.737 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:12:24.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:12:24.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:24.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:25.095 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/128005,v1:127.0.0.1:6803/128005] [v2:127.0.0.1:6804/128005,v1:127.0.0.1:6805/128005] exists,up bd81369b-cbf0-4563-8f75-2b6575d4c8ae 2024-09-15T03:12:25.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:12:25.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:12:25.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:12:25.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-09-15T03:12:25.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:12:25.490 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:12:25.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:12:25.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:12:26.009 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:12:26.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:12:27.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:12:28.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-09-15T03:12:28.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:12:28.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:12:28.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:12:28.208 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:12:28.209 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:12:28.209 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:12:28.209 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:12:28.209 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:12:28.209 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:12:28.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:12:28.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:12:28.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:12:28.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:12:28.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:12:28.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:12:28.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:12:28.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:12:28.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:28.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:12:28.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836483 2024-09-15T03:12:28.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836483 2024-09-15T03:12:28.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836483' 2024-09-15T03:12:28.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:28.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836483 2024-09-15T03:12:28.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:28.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:12:28.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836483 2024-09-15T03:12:28.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:28.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836483 2024-09-15T03:12:28.758 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836483 2024-09-15T03:12:28.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-09-15T03:12:28.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:12:29.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836481 -lt 21474836483 2024-09-15T03:12:29.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:12:30.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:12:30.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:12:30.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836483 2024-09-15T03:12:30.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:12:30.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:30.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:30.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:12:30.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:12:30.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:12:30.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:12:30.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:12:30.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:12:30.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:12:30.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:12:31.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:12:31.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:12:31.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:31.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:31.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:12:31.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:12:31.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:12:31.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-15T03:12:31.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:12:31.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:12:32.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:12:32.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-09-15T03:12:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-15T03:12:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:12:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-15T03:12:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:12:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-15T03:12:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:12:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:12:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:12:33.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:12:33.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:12:33.781 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.0 2024-09-15T03:12:33.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:12:33.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-15T03:12:33.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:12:33.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:12:33.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:12:33.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:12:33.803 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:33.801+0000 7f611a94f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:33.805 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:33.804+0000 7f611a94f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:33.806 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:33.805+0000 7f611a94f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:12:34.137 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-09-15T03:12:34.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:34.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:34.336 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:34.335+0000 7f611a94f440 -1 Falling back to public interface 2024-09-15T03:12:34.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:35.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:35.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:35.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:12:35.487 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T03:12:35.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:35.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:35.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:36.751 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:36.750+0000 7f611a94f440 -1 osd.0 13 log_to_monitors true 2024-09-15T03:12:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:12:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-09-15T03:12:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:37.423 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 up in weight 1 up_from 16 up_thru 7 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/129641,v1:127.0.0.1:6803/129641] [v2:127.0.0.1:6804/129641,v1:127.0.0.1:6805/129641] exists,up bd81369b-cbf0-4563-8f75-2b6575d4c8ae 2024-09-15T03:12:37.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:12:37.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:12:37.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:12:37.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:12:37.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:12:37.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:12:37.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:12:37.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:12:37.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:12:37.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:12:37.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:37.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:12:38.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476739 2024-09-15T03:12:38.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476739 2024-09-15T03:12:38.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-68719476739' 2024-09-15T03:12:38.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:38.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-68719476739 2024-09-15T03:12:38.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:38.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:12:38.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-68719476739 2024-09-15T03:12:38.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:38.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476739 2024-09-15T03:12:38.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2024-09-15T03:12:38.068 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 68719476739 2024-09-15T03:12:38.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:12:38.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 68719476739 2024-09-15T03:12:38.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:12:39.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:12:39.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:12:39.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476739 -lt 68719476739 2024-09-15T03:12:39.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:12:39.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:39.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:12:40.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:12:40.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:12:40.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:12:40.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:40.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-15T03:12:41.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:12:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-15T03:12:41.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-09-15T03:12:42.599 INFO:tasks.workunit.client.0.smithi137.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-09-15T03:12:42.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-15T03:12:42.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:12:42.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:12:42.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:12:42.972 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:12:42.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:12:42.973 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:12:42.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:12:42.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:12:42.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:12:42.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:12:42.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:12:42.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:12:42.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:12:42.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:12:42.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:12:42.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:12:42.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:12:42.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:42.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:42.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:12:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:12:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:12:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:12:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_pg_scrub td/ceph-helpers 2024-09-15T03:12:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1902: test_pg_scrub: local dir=td/ceph-helpers 2024-09-15T03:12:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1904: test_pg_scrub: setup td/ceph-helpers 2024-09-15T03:12:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:12:42.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:12:42.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:12:42.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:12:42.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:12:42.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:12:42.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:12:42.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:12:42.992 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:12:42.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:12:42.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:12:42.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:12:42.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:12:42.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:12:42.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:12:42.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:12:42.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:12:42.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:42.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:42.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:12:42.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:12:42.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:12:42.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:12:43.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:12:43.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:43.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:43.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:12:43.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:12:43.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:12:43.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:12:43.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:12:43.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:12:43.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:12:43.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:12:43.367 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:12:43.368 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:43.368 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:43.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:12:43.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:12:43.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:12:43.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:12:43.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:12:43.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:12:43.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:43.595 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:12:43.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:12:43.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:12:43.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:12:43.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:12:43.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:12:43.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:12:43.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:12:43.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:12:43.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:12:43.681 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:12:43.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:12:43.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:43.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:43.681 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:12:43.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:12:43.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:12:43.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: test_pg_scrub: run_mgr td/ceph-helpers x 2024-09-15T03:12:43.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:12:43.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:12:43.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:12:43.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:12:43.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:12:43.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:12:44.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:12:44.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:12:44.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:12:44.594 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:12:44.594 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:44.594 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:44.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:12:44.595 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:12:44.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:12:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: run_osd td/ceph-helpers 0 2024-09-15T03:12:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:12:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:12:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:12:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:12:44.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:12:44.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:12:44.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:12:44.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:12:44.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:12:44.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:12:44.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:12:44.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:12:44.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:12:44.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=15a14486-94a5-473f-9aa7-e4439615f050 2024-09-15T03:12:44.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 15a14486-94a5-473f-9aa7-e4439615f050' 2024-09-15T03:12:44.625 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 15a14486-94a5-473f-9aa7-e4439615f050 2024-09-15T03:12:44.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:12:44.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsUOZmwjXiJRAAAXYAdkJAinUnsSxFXV+qng== 2024-09-15T03:12:44.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsUOZmwjXiJRAAAXYAdkJAinUnsSxFXV+qng=="}' 2024-09-15T03:12:44.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 15a14486-94a5-473f-9aa7-e4439615f050 -i td/ceph-helpers/0/new.json 2024-09-15T03:12:44.947 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:12:44.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:12:44.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsUOZmwjXiJRAAAXYAdkJAinUnsSxFXV+qng== --osd-uuid 15a14486-94a5-473f-9aa7-e4439615f050 2024-09-15T03:12:44.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:44.981+0000 7f419b647440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:45.001 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:45.001+0000 7f419b647440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:45.002 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:45.002+0000 7f419b647440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:45.003 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:45.002+0000 7f419b647440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:12:45.003 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:45.002+0000 7f419b647440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:12:45.003 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:45.002+0000 7f419b647440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:12:45.042 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:45.041+0000 7f419b647440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:12:50.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:12:50.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:12:50.683 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:12:50.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:12:50.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:12:51.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:12:51.165 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:12:51.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:12:51.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:12:51.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:12:51.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:12:51.186 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:51.186+0000 7fe50ea0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:51.188 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:51.188+0000 7fe50ea0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:51.189 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:51.189+0000 7fe50ea0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:51.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:12:51.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:12:51.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:12:51.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:12:51.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:12:51.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:51.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:12:51.515 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:12:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:51.705 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:51.704+0000 7fe50ea0a440 -1 Falling back to public interface 2024-09-15T03:12:51.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:52.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:52.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:52.890 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:12:52.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:12:52.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:52.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:53.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:54.213 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:12:54.212+0000 7fe50ea0a440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:12:54.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:54.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:54.248 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:12:54.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:12:54.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:54.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:54.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:55.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:55.637 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:12:55.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:55.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:12:55.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:55.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:56.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:57.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:57.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:57.051 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:12:57.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:12:57.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:57.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:57.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:58.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:58.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:58.410 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:12:58.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:12:58.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:58.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:12:58.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:59.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:59.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:59.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:12:59.781 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:12:59.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:59.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:00.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:01.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:01.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:01.189 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:13:01.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:13:01.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:01.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:01.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:02.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:02.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:02.590 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:13:02.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:13:02.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:02.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:03.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:04.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:04.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:04.097 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:13:04.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:13:04.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:04.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:04.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:05.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:05.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:05.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:13:05.502 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:13:05.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:05.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:06.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:07.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:07.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:07.047 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:13:07.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:13:07.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:07.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:07.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:08.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:08.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:08.530 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:13:08.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:13:08.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:08.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:09.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:10.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:10.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:10.261 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:13:10.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:13:10.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:10.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:11.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:12.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:12.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:12.342 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:13:12.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:13:12.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:12.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:13.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:14.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:14.194 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:13:14.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:13:14.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:14.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:15.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:16.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:16.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:16.532 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:13:16.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:13:16.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:16.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:17.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:18.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:18.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:18.109 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:13:18.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:13:18.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:18.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:18.479 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/131907,v1:127.0.0.1:6803/131907] [v2:127.0.0.1:6804/131907,v1:127.0.0.1:6805/131907] exists,up 15a14486-94a5-473f-9aa7-e4439615f050 2024-09-15T03:13:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:13:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:13:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:13:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_pg_scrub: create_rbd_pool 2024-09-15T03:13:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:13:18.819 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:13:18.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:13:18.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:13:19.298 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:13:19.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:13:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:13:21.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1909: test_pg_scrub: wait_for_clean 2024-09-15T03:13:21.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:13:21.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:13:21.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:13:21.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:13:21.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:13:21.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:13:21.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:13:21.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:13:21.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:13:21.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:13:21.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:13:21.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:13:21.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:13:21.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:13:21.544 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:13:21.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:13:21.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:13:21.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:13:21.890 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:13:21.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-15T03:13:21.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-15T03:13:21.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-15T03:13:21.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:13:21.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-15T03:13:21.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:13:21.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:13:21.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-15T03:13:21.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:13:21.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-15T03:13:21.999 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836485 2024-09-15T03:13:22.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-15T03:13:22.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:13:22.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-15T03:13:22.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:13:22.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:13:22.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:13:22.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:13:22.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:13:22.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:13:22.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:13:22.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:13:22.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:13:22.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:13:22.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:13:23.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:13:23.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:13:23.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:13:23.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: test_pg_scrub: pg_scrub 1.0 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:13:23.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:13:24.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-15T03:13:19.123247+0000 2024-09-15T03:13:24.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-15T03:13:24.368 INFO:tasks.workunit.client.0.smithi137.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-15T03:13:24.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-15T03:13:19.123247+0000 2024-09-15T03:13:24.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-15T03:13:19.123247+0000 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:13:24.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:13:24.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:13:19.123247+0000 '>' 2024-09-15T03:13:19.123247+0000 2024-09-15T03:13:24.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:13:25.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:13:25.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:13:25.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:13:25.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:13:25.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:13:25.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:13:25.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:13:26.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:13:19.123247+0000 '>' 2024-09-15T03:13:19.123247+0000 2024-09-15T03:13:26.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:13:27.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:13:27.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:13:27.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:13:27.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:13:27.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:13:27.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:13:27.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:13:27.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:13:19.123247+0000 '>' 2024-09-15T03:13:19.123247+0000 2024-09-15T03:13:27.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:13:28.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:13:28.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:13:28.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:13:28.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:13:28.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:13:28.456 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:13:28.456 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:13:28.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:13:26.219623+0000 '>' 2024-09-15T03:13:19.123247+0000 2024-09-15T03:13:28.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-15T03:13:28.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-15T03:13:28.805 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:13:28.805 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:13:28.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:13:28.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:13:28.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_pg_scrub: TIMEOUT=1 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_pg_scrub: pg_scrub 1.0 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:13:28.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:13:29.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-15T03:13:26.219623+0000 2024-09-15T03:13:29.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-15T03:13:29.579 INFO:tasks.workunit.client.0.smithi137.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-15T03:13:29.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-15T03:13:26.219623+0000 2024-09-15T03:13:29.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-15T03:13:29.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-15T03:13:26.219623+0000 2024-09-15T03:13:29.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T03:13:29.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-15T03:13:29.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-15T03:13:29.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:13:29.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:13:29.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:13:29.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:13:29.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:13:29.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:13:26.219623+0000 '>' 2024-09-15T03:13:26.219623+0000 2024-09-15T03:13:29.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:13:30.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: wait_for_scrub: return 1 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1913: test_pg_scrub: teardown td/ceph-helpers 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:13:30.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:13:31.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:13:31.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:13:31.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:13:31.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:13:31.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:13:31.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:13:31.040 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:13:31.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:31.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:13:31.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:13:31.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:31.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:13:31.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:13:31.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:13:31.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:13:31.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:31.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:31.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:13:31.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:13:31.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:13:31.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:13:31.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_repair td/ceph-helpers 2024-09-15T03:13:31.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1862: test_repair: local dir=td/ceph-helpers 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: setup td/ceph-helpers 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:13:31.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:13:31.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:13:31.054 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:13:31.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:13:31.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:13:31.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:13:31.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:13:31.056 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:13:31.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:31.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:13:31.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:13:31.058 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:31.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:13:31.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:13:31.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:13:31.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:13:31.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:31.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:31.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:13:31.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:13:31.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:13:31.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:13:31.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:13:31.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:31.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:31.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:13:31.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:13:31.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:13:31.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:31.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:31.423 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:31.423 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:31.423 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:31.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:13:31.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:13:31.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:13:31.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:13:31.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:13:31.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:13:31.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:13:31.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:13:31.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:13:31.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:13:31.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:13:31.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:13:31.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:13:31.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:13:31.749 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:13:31.749 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:13:31.750 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:13:31.750 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:13:31.750 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:31.750 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:31.750 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:13:31.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:13:31.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:13:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_repair: run_mgr td/ceph-helpers x 2024-09-15T03:13:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:13:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:13:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:13:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:13:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:13:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:13:32.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:13:32.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:32.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:32.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:32.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:32.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:32.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:13:32.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:13:32.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:13:32.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_repair: run_osd td/ceph-helpers 0 2024-09-15T03:13:32.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:13:32.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:13:32.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:13:32.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:13:32.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:13:32.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:13:32.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:13:32.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:13:32.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:13:32.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:13:32.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:13:32.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d2f5844f-3f8d-4d9c-897c-5809b7121e6c 2024-09-15T03:13:32.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d2f5844f-3f8d-4d9c-897c-5809b7121e6c' 2024-09-15T03:13:32.698 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 d2f5844f-3f8d-4d9c-897c-5809b7121e6c 2024-09-15T03:13:32.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:13:32.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcUOZmBlg7KhAAZXeonyDK+qCYtrzaiu7dYA== 2024-09-15T03:13:32.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcUOZmBlg7KhAAZXeonyDK+qCYtrzaiu7dYA=="}' 2024-09-15T03:13:32.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d2f5844f-3f8d-4d9c-897c-5809b7121e6c -i td/ceph-helpers/0/new.json 2024-09-15T03:13:33.003 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:13:33.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:13:33.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcUOZmBlg7KhAAZXeonyDK+qCYtrzaiu7dYA== --osd-uuid d2f5844f-3f8d-4d9c-897c-5809b7121e6c 2024-09-15T03:13:33.037 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:33.037+0000 7f2f64e17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:33.066 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:33.065+0000 7f2f64e17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:33.067 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:33.066+0000 7f2f64e17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:33.067 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:33.066+0000 7f2f64e17440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:13:33.067 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:33.066+0000 7f2f64e17440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:13:33.067 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:33.066+0000 7f2f64e17440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:13:33.106 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:33.105+0000 7f2f64e17440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:13:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:13:38.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:13:38.766 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:13:38.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:13:38.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:13:39.248 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:13:39.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:13:39.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:13:39.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:13:39.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:13:39.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:13:39.270 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:39.269+0000 7f00c1721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:39.272 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:39.271+0000 7f00c1721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:39.273 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:39.272+0000 7f00c1721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:39.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:13:39.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:13:39.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:13:39.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:13:39.596 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:13:39.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:13:39.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:39.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:13:39.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:39.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:39.792 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:39.791+0000 7f00c1721440 -1 Falling back to public interface 2024-09-15T03:13:39.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:40.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:40.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:40.953 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:13:40.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:13:40.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:40.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:41.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:42.316 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:13:42.315+0000 7f00c1721440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:13:42.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:42.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:42.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:13:42.354 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:13:42.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:42.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:42.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:43.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:43.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:43.708 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:13:43.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:13:43.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:43.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:44.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:45.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:45.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:45.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:13:45.118 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:13:45.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:45.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:45.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:46.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:46.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:46.519 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:13:46.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:13:46.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:46.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:46.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:48.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:48.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:48.001 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:13:48.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:13:48.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:48.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:48.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:49.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:49.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:49.380 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:13:49.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:13:49.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:49.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:49.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:50.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:50.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:50.769 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:13:50.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:13:50.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:50.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:51.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:52.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:52.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:52.132 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:13:52.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:13:52.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:52.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:52.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:53.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:53.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:53.538 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:13:53.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:13:53.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:53.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:54.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:55.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:55.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:55.033 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:13:55.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:13:55.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:55.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:55.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:56.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:56.457 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:13:56.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:56.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:13:56.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:56.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:57.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:58.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:58.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:58.533 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:13:58.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:13:58.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:58.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:59.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:00.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:00.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:00.432 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:14:00.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:14:00.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:00.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:01.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:02.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:02.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:02.191 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:14:02.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:14:02.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:02.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:03.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:04.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:04.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:04.256 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:14:04.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:14:04.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:04.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:04.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:05.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:05.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:05.686 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:14:05.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:14:05.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:05.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:06.034 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/134429,v1:127.0.0.1:6803/134429] [v2:127.0.0.1:6804/134429,v1:127.0.0.1:6805/134429] exists,up d2f5844f-3f8d-4d9c-897c-5809b7121e6c 2024-09-15T03:14:06.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:14:06.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:14:06.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:14:06.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_repair: create_rbd_pool 2024-09-15T03:14:06.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:14:06.386 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:14:06.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:14:06.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:14:06.883 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:14:06.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:14:07.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:14:09.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_repair: wait_for_clean 2024-09-15T03:14:09.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:14:09.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:14:09.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:14:09.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:14:09.134 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:14:09.134 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:14:09.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:14:09.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:14:09.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:14:09.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:14:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:14:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:14:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:14:09.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:14:09.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:14:09.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:14:09.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:14:09.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:14:09.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:14:09.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836488 2024-09-15T03:14:09.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836488 2024-09-15T03:14:09.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488' 2024-09-15T03:14:09.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:14:09.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836488 2024-09-15T03:14:09.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:14:09.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:14:09.684 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836488 2024-09-15T03:14:09.684 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:14:09.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836488 2024-09-15T03:14:09.685 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836488 2024-09-15T03:14:09.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-15T03:14:09.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:14:10.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-15T03:14:10.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:14:11.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:14:11.040 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:14:11.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-15T03:14:11.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:14:11.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:14:11.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:14:11.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:14:11.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:14:11.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:14:11.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:14:11.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:14:11.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:14:11.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:14:11.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:14:12.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:14:12.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:14:12.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:14:12.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:14:12.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_repair: repair 1.0 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:14:12.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:14:13.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-15T03:14:06.705479+0000 2024-09-15T03:14:13.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-15T03:14:13.423 INFO:tasks.workunit.client.0.smithi137.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-15T03:14:06.705479+0000 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-15T03:14:06.705479+0000 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:14:13.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:14:13.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:14:13.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:14:13.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:14:13.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:14:06.705479+0000 '>' 2024-09-15T03:14:06.705479+0000 2024-09-15T03:14:13.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:14:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:14:14.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:14:14.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:14:14.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:14:14.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:14:14.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:14:14.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:14:15.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:14:06.705479+0000 '>' 2024-09-15T03:14:06.705479+0000 2024-09-15T03:14:15.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:14:16.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:14:16.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:14:16.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:14:16.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:14:16.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:14:16.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:14:16.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:14:15.130300+0000 '>' 2024-09-15T03:14:06.705479+0000 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:14:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:14:16.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: TIMEOUT=1 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: repair 1.0 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:14:16.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:14:16.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-15T03:14:15.130300+0000 2024-09-15T03:14:16.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-15T03:14:17.255 INFO:tasks.workunit.client.0.smithi137.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-15T03:14:17.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-15T03:14:15.130300+0000 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-15T03:14:15.130300+0000 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:14:17.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:14:17.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:14:15.130300+0000 '>' 2024-09-15T03:14:15.130300+0000 2024-09-15T03:14:17.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: wait_for_scrub: return 1 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_repair: teardown td/ceph-helpers 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:14:18.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:14:18.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:14:18.715 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:14:18.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:14:18.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:14:18.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:14:18.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:14:18.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:14:18.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:14:18.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:14:18.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:14:18.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:14:18.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:14:18.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:14:18.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:14:18.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:14:18.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:18.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:18.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_run_in_background td/ceph-helpers 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: test_run_in_background: local pids 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: test_run_in_background: run_in_background pids sleep 1 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 135870"' 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 135870' 2024-09-15T03:14:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: test_run_in_background: run_in_background pids sleep 1 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 135872"' 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sleep 1 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 135872' 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/135873: /' 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sleep 1 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/135876: /' 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_run_in_background: echo 135870 135872 2024-09-15T03:14:18.732 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_run_in_background: wc -w 2024-09-15T03:14:18.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_run_in_background: test 2 = 2 2024-09-15T03:14:18.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: test_run_in_background: wait 135870 135872 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_run_mon td/ceph-helpers 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:14:19.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:14:19.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:14:19.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:14:19.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:14:19.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:14:19.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:14:19.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:14:19.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:14:19.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:14:19.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:14:19.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:14:19.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:14:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:14:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:14:19.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:14:19.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:14:19.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:14:19.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:14:19.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:14:19.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:14:19.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:14:19.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:14:19.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:14:19.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:19.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:19.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:14:19.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:14:19.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:14:19.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:14:19.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:14:19.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:19.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:19.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:14:19.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:14:19.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:14:19.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T03:14:20.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:14:20.108 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:14:20.108 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:14:20.108 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:14:20.108 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:20.108 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:20.108 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:14:20.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:14:20.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:14:20.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:14:20.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:14:20.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:14:20.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:14:20.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:14:20.378 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:14:20.378 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:14:20.378 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:14:20.379 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:14:20.379 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:20.379 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:20.379 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:14:20.379 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:14:20.379 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:14:20.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:14:20.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:20.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:14:20.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:14:20.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:14:20.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-09-15T03:14:20.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-09-15T03:14:21.317 INFO:tasks.workunit.client.0.smithi137.stderr:dumped monmap epoch 1 2024-09-15T03:14:21.326 INFO:tasks.workunit.client.0.smithi137.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-09-15T03:14:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-09-15T03:14:21.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:14:21.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:14:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:14:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:14:21.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:14:21.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:14:21.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-09-15T03:14:21.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:14:21.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:14:21.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:14:21.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:14:21.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:14:21.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-09-15T03:14:21.651 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:21.649+0000 7f594e23b900 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-15T03:14:21.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:14:21.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:14:21.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:14:21.653 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:14:21.653 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:21.653 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:21.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:14:21.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T03:14:21.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:14:21.874 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:21.875 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:21.875 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:14:21.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:14:21.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:14:21.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:14:21.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:14:21.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:14:21.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:14:21.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:14:21.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:14:22.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:14:22.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:14:22.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:14:22.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:14:22.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:14:22.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:14:22.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:14:22.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:14:22.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:14:22.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:14:22.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:14:22.029 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:14:22.029 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:22.029 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:22.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:14:22.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:14:22.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:14:22.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:14:22.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:14:22.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:14:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:14:22.032 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:14:22.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=848a670e-9dd6-4a9e-bee4-c4de8164afa3 2024-09-15T03:14:22.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 848a670e-9dd6-4a9e-bee4-c4de8164afa3' 2024-09-15T03:14:22.032 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 848a670e-9dd6-4a9e-bee4-c4de8164afa3 2024-09-15T03:14:22.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:14:22.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOUeZm9bRxAhAAbyRzmLGgQOECExlzks7/ew== 2024-09-15T03:14:22.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOUeZm9bRxAhAAbyRzmLGgQOECExlzks7/ew=="}' 2024-09-15T03:14:22.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 848a670e-9dd6-4a9e-bee4-c4de8164afa3 -i td/ceph-helpers/0/new.json 2024-09-15T03:14:22.331 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:14:22.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:14:22.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOUeZm9bRxAhAAbyRzmLGgQOECExlzks7/ew== --osd-uuid 848a670e-9dd6-4a9e-bee4-c4de8164afa3 2024-09-15T03:14:22.365 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:22.363+0000 7f99ffc45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:22.378 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:22.377+0000 7f99ffc45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:22.379 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:22.378+0000 7f99ffc45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:22.380 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:22.378+0000 7f99ffc45440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:14:22.380 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:22.378+0000 7f99ffc45440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:14:22.380 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:22.378+0000 7f99ffc45440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:14:22.428 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:22.426+0000 7f99ffc45440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:14:27.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:14:27.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:14:27.733 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:14:27.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:14:27.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:14:27.927 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:14:27.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:14:27.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:14:27.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:14:27.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:14:27.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:14:27.950 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:27.948+0000 7fde3d618440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:27.956 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:27.955+0000 7fde3d618440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:27.957 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:27.956+0000 7fde3d618440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:28.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:14:28.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:14:28.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:14:28.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:14:28.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:14:28.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:28.126 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:14:28.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:14:28.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:28.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:28.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:28.486 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:28.485+0000 7fde3d618440 -1 Falling back to public interface 2024-09-15T03:14:29.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:29.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:29.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:14:29.337 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:14:29.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:29.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:29.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:30.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:30.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:30.526 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:14:30.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:14:30.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:30.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:30.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:31.082 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:31.081+0000 7fde3d618440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:14:31.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:31.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:31.714 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:14:31.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:14:31.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:31.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:32.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:33.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:33.050 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:14:33.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:14:33.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:33.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:33.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:34.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:34.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:34.240 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:14:34.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:14:34.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:34.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:34.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:35.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:35.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:35.428 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:14:35.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:14:35.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:35.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:35.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:36.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:36.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:36.618 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:14:36.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:14:36.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:36.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:36.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:37.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:37.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:37.806 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:14:37.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:14:37.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:37.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:37.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:38.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:38.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:38.992 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:14:38.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:14:38.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:38.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:39.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:40.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:40.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:40.180 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:14:40.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:14:40.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:40.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:40.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:41.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:41.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:41.368 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:14:41.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:14:41.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:41.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:41.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:42.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:42.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:42.557 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:14:42.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:14:42.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:42.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:42.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:43.741 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:14:43.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:14:43.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:43.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:43.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:44.929 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:14:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:14:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:45.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:46.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:46.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:46.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:14:46.117 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:14:46.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:46.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:46.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:47.305 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:14:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:14:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:47.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:47.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:48.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:48.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:48.494 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:14:48.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:14:48.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:48.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:48.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:49.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:49.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:49.681 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:14:49.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:14:49.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:49.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:49.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:50.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:50.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:50.869 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:14:50.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:14:50.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:50.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:51.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:52.058 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:14:52.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:14:52.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:52.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:14:52.339 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/136904,v1:127.0.0.1:6801/136904] [v2:127.0.0.1:6802/136904,v1:127.0.0.1:6803/136904] exists,up 848a670e-9dd6-4a9e-bee4-c4de8164afa3 2024-09-15T03:14:52.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:14:52.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:14:52.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:14:52.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-09-15T03:14:52.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:14:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:14:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:14:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:14:52.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:14:52.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae05e9f8-c39d-44ab-830d-28e3d3f17fbb 2024-09-15T03:14:52.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ae05e9f8-c39d-44ab-830d-28e3d3f17fbb' 2024-09-15T03:14:52.347 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 ae05e9f8-c39d-44ab-830d-28e3d3f17fbb 2024-09-15T03:14:52.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:14:52.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsUeZm9aZGFRAANzjhJcGdwk5pETPub+vQpA== 2024-09-15T03:14:52.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsUeZm9aZGFRAANzjhJcGdwk5pETPub+vQpA=="}' 2024-09-15T03:14:52.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae05e9f8-c39d-44ab-830d-28e3d3f17fbb -i td/ceph-helpers/1/new.json 2024-09-15T03:14:52.702 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:14:52.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:14:52.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsUeZm9aZGFRAANzjhJcGdwk5pETPub+vQpA== --osd-uuid ae05e9f8-c39d-44ab-830d-28e3d3f17fbb 2024-09-15T03:14:52.736 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:52.734+0000 7f14cbd1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:52.757 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:52.756+0000 7f14cbd1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:52.758 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:52.757+0000 7f14cbd1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:52.759 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:52.757+0000 7f14cbd1f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:14:52.759 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:52.757+0000 7f14cbd1f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:14:52.759 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:52.757+0000 7f14cbd1f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:14:52.797 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:52.796+0000 7f14cbd1f440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:14:58.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:14:58.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:14:58.387 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:14:58.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:14:58.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:14:58.582 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:14:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:14:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:14:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:14:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:14:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:14:58.604 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:58.603+0000 7fc48af21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:58.624 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:58.623+0000 7fc48af21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:58.625 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:58.624+0000 7fc48af21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:58.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:14:58.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:14:58.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:14:58.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:14:58.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:14:58.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:58.795 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:14:58.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:14:58.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:58.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:59.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:59.156 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:14:59.155+0000 7fc48af21440 -1 Falling back to public interface 2024-09-15T03:15:00.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:00.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:00.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:15:00.006 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:15:00.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:00.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:00.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:01.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:01.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:01.186 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:15:01.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:15:01.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:01.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:01.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:01.743 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:01.741+0000 7fc48af21440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:15:02.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:02.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:02.374 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:15:02.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:15:02.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:02.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:02.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:03.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:03.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:03.595 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:15:03.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:15:03.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:03.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:03.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:04.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:04.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:04.784 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:15:04.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:15:04.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:04.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:04.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:05.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:05.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:05.973 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:15:05.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:15:05.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:05.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:06.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:07.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:07.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:07.161 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:15:07.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:15:07.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:07.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:07.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:08.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:08.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:08.349 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:15:08.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:15:08.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:08.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:08.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:09.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:09.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:09.540 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:15:09.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:15:09.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:09.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:09.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:10.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:10.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:10.729 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:15:10.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:15:10.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:10.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:10.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:11.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:11.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:11.916 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:15:11.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:15:11.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:11.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:12.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:13.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:13.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:13.103 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:15:13.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:15:13.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:13.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:13.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:14.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:14.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:14.293 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:15:14.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:15:14.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:14.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:14.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:15.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:15.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:15.482 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:15:15.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:15:15.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:15.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:15.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:16.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:16.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:16.670 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:15:16.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:15:16.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:16.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:16.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:17.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:17.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:17.861 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:15:17.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:15:17.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:17.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:18.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:19.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:19.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:19.049 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:15:19.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:15:19.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:19.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:19.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:20.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:20.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:20.238 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:15:20.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:15:20.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:20.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:20.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:21.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:21.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:21.429 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:15:21.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:15:21.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:21.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:22.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:22.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:22.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:15:22.618 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:15:22.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:22.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:15:22.832 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/138734,v1:127.0.0.1:6809/138734] [v2:127.0.0.1:6810/138734,v1:127.0.0.1:6811/138734] exists,up ae05e9f8-c39d-44ab-830d-28e3d3f17fbb 2024-09-15T03:15:22.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:15:22.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:15:22.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:15:22.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-09-15T03:15:22.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:15:22.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:15:22.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:15:22.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:15:22.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-15T03:15:22.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:15:22.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:15:22.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:15:22.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:15:22.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-15T03:15:22.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-15T03:15:22.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:15:22.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:15:22.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:15:22.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:15:22.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:15:22.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:15:22.839 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:15:22.839 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:15:22.839 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:15:22.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:15:22.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:15:22.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-15T03:15:22.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:15:22.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=40fd5e5f-0294-413d-a813-db2a00b6611c 2024-09-15T03:15:22.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 40fd5e5f-0294-413d-a813-db2a00b6611c' 2024-09-15T03:15:22.841 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 40fd5e5f-0294-413d-a813-db2a00b6611c 2024-09-15T03:15:22.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:15:22.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKUeZmv8qdMhAAOGVUIHVdVeBw75yrFB6KEg== 2024-09-15T03:15:22.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKUeZmv8qdMhAAOGVUIHVdVeBw75yrFB6KEg=="}' 2024-09-15T03:15:22.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 40fd5e5f-0294-413d-a813-db2a00b6611c -i td/ceph-helpers/2/new.json 2024-09-15T03:15:23.170 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:15:23.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-15T03:15:23.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKUeZmv8qdMhAAOGVUIHVdVeBw75yrFB6KEg== --osd-uuid 40fd5e5f-0294-413d-a813-db2a00b6611c 2024-09-15T03:15:23.205 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:23.204+0000 7f7d01a48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:23.227 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:23.225+0000 7f7d01a48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:23.226+0000 7f7d01a48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:23.226+0000 7f7d01a48440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:15:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:23.226+0000 7f7d01a48440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:15:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:23.226+0000 7f7d01a48440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:15:23.268 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:23.266+0000 7f7d01a48440 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-15T03:15:29.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-15T03:15:29.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:15:29.106 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:15:29.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:15:29.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:15:29.300 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:15:29.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:15:29.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:15:29.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:15:29.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:15:29.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:15:29.323 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:29.322+0000 7fce37408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:29.328 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:29.328+0000 7fce37408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:29.329 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:29.329+0000 7fce37408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:29.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:15:29.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:15:29.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:15:29.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:15:29.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:15:29.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:29.502 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:15:29.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:15:29.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:29.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:29.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:29.857 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:29.857+0000 7fce37408440 -1 Falling back to public interface 2024-09-15T03:15:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:15:30.708 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:15:30.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:30.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:30.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:31.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:31.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:31.890 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:15:31.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:15:31.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:31.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:32.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:32.464 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:32.463+0000 7fce37408440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:15:33.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:33.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:33.071 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:15:33.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:15:33.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:33.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:33.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:34.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:34.341 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:15:34.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:34.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:15:34.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:34.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:34.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:35.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:35.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:35.523 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:15:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:15:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:35.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:36.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:36.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:36.711 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:15:36.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:15:36.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:36.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:36.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:37.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:37.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:37.901 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:15:37.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:15:37.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:37.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:38.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:39.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:39.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:39.089 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:15:39.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:15:39.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:39.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:39.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:40.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:40.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:40.275 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:15:40.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:15:40.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:40.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:40.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:41.463 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:15:41.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:15:41.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:41.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:41.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:42.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:42.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:42.650 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:15:42.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:15:42.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:42.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:42.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:43.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:43.838 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:15:43.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:43.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:15:43.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:43.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:44.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:45.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:45.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:45.022 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:15:45.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:15:45.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:45.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:45.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:46.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:46.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:46.209 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:15:46.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:15:46.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:46.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:46.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:47.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:47.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:47.395 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:15:47.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:15:47.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:47.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:47.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:48.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:48.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:48.585 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:15:48.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:15:48.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:48.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:48.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:49.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:49.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:49.773 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:15:49.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:15:49.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:49.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:49.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:50.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:50.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:15:50.961 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:15:50.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:50.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:51.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:52.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:52.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:52.148 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:15:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:15:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:52.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:53.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:53.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:53.337 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:15:53.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:15:53.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:53.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:53.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:54.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:54.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:54.556 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-09-15T03:15:54.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-15T03:15:54.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:54.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:54.740 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/140563,v1:127.0.0.1:6817/140563] [v2:127.0.0.1:6818/140563,v1:127.0.0.1:6819/140563] exists,up 40fd5e5f-0294-413d-a813-db2a00b6611c 2024-09-15T03:15:54.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:15:54.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:15:54.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:15:54.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-09-15T03:15:54.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:15:54.950 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:15:54.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:15:54.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:15:55.299 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:15:55.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:15:56.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:15:58.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-09-15T03:15:58.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-09-15T03:15:58.610 INFO:tasks.workunit.client.0.smithi137.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 24 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-15T03:15:58.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-09-15T03:15:58.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:15:58.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:15:58.611 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:15:58.611 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:15:58.612 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:15:58.612 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:15:58.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-09-15T03:15:58.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get osd_pool_default_size 2024-09-15T03:15:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-09-15T03:15:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-09-15T03:15:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-09-15T03:15:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-09-15T03:15:58.750 INFO:tasks.workunit.client.0.smithi137.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-15T03:15:58.750 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.748+0000 7fe2f7eaf640 -1 failed for service _ceph-mon._tcp 2024-09-15T03:15:58.750 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.748+0000 7fe2f7eaf640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-15T03:15:58.750 INFO:tasks.workunit.client.0.smithi137.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-09-15T03:15:58.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-09-15T03:15:58.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-09-15T03:15:58.817 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.815+0000 7f077575d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-15T03:15:58.817 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.815+0000 7f077575d640 -1 AuthRegistry(0x7f0770064520) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-15T03:15:58.818 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.816+0000 7f077575d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-15T03:15:58.818 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.816+0000 7f077575d640 -1 AuthRegistry(0x7f077575bff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-15T03:15:58.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.817+0000 7f077575d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-15T03:15:58.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.817+0000 7f077575d640 -1 AuthRegistry(0x7f0770061630) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-15T03:15:58.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.817+0000 7f077575d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-15T03:15:58.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.817+0000 7f077575d640 -1 AuthRegistry(0x7f07700f6290) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-15T03:15:58.820 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.817+0000 7f077575d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-15T03:15:58.820 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:15:58.817+0000 7f077575d640 -1 AuthRegistry(0x7f0770061630) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: cluster: 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: id: 145a3ccf-df94-4947-9f60-2f1acb4777f7 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: health: HEALTH_OK 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: services: 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: mon: 1 daemons, quorum a (age 96s) 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: mgr: no daemons active 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: osd: 3 osds: 3 up (since 5s), 3 in (since 35s) 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: data: 2024-09-15T03:15:58.918 INFO:tasks.workunit.client.0.smithi137.stdout: pools: 0 pools, 0 pgs 2024-09-15T03:15:58.919 INFO:tasks.workunit.client.0.smithi137.stdout: objects: 0 objects, 0 B 2024-09-15T03:15:58.919 INFO:tasks.workunit.client.0.smithi137.stdout: usage: 0 B used, 0 B / 0 B avail 2024-09-15T03:15:58.919 INFO:tasks.workunit.client.0.smithi137.stdout: pgs: 2024-09-15T03:15:58.919 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-09-15T03:15:58.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-09-15T03:15:58.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:15:58.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:15:58.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:15:58.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:15:58.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:16:04.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:16:04.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:16:04.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:16:04.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:16:04.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:16:04.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:16:04.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:16:04.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:16:04.260 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:04.259+0000 7f903a828900 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-15T03:16:04.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:16:04.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:04.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:04.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:04.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:04.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:04.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:04.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:16:04.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:16:04.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:16:04.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:04.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:04.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:16:04.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:16:04.522 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:04.522 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:04.522 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:04.522 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:04.522 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:04.522 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:04.523 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:04.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:04.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:16:04.603 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:04.604 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:04.605 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:04.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:04.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:16:04.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-09-15T03:16:04.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:04.680 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:04.680 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:04.680 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:04.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:04.681 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:04.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-09-15T03:16:04.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get osd_pool_default_size 2024-09-15T03:16:04.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-09-15T03:16:04.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-09-15T03:16:04.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-09-15T03:16:04.753 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:16:04.753 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:16:04.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:16:04.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:16:04.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:16:04.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T03:16:04.877 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:04.875+0000 7ff339ed9900 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-15T03:16:04.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:16:04.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:04.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:04.879 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:04.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:04.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:04.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:04.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:16:05.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:16:05.142 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:05.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:05.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:16:05.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:16:05.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:05.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:05.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:16:05.293 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-09-15T03:16:05.293 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:05.293 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:05.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:05.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:05.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:05.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:05.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-09-15T03:16:05.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get osd_pool_default_size 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:16:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:16:05.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:16:05.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-09-15T03:16:05.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:16:05.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:16:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:16:05.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:16:05.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:16:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:16:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:16:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:16:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:16:05.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:16:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:16:05.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:16:05.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:16:05.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:16:05.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:16:05.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:05.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:16:05.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:16:05.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:05.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:16:05.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:16:05.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:16:05.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:16:05.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:05.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:05.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:16:05.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:16:05.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:16:05.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:16:05.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_run_osd td/ceph-helpers 2024-09-15T03:16:05.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:16:05.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:16:05.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:16:05.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:16:05.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:16:05.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:16:05.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:16:05.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:16:05.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:16:05.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:05.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:16:05.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:16:05.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:05.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:16:05.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:16:05.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:16:05.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:16:05.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:05.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:05.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:16:05.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:16:05.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:16:05.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:16:05.706 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:16:05.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:05.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:05.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:16:05.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:16:05.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-15T03:16:06.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:16:06.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:06.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:06.104 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:06.104 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:06.104 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:06.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:06.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:16:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:06.373 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:06.374 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:06.374 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:06.374 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:06.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:06.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:06.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:06.449 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:06.450 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:06.450 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:06.450 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:16:06.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:06.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:16:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-09-15T03:16:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:16:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:16:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:16:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:16:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:16:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:16:07.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:16:07.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:07.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:07.366 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:07.366 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:07.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:07.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:16:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:16:07.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:16:07.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:16:07.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:16:07.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:16:07.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:16:07.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:16:07.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c2d95d96-59e8-4a90-a9b8-958bac6836b8 2024-09-15T03:16:07.397 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 c2d95d96-59e8-4a90-a9b8-958bac6836b8 2024-09-15T03:16:07.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c2d95d96-59e8-4a90-a9b8-958bac6836b8' 2024-09-15T03:16:07.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:16:07.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3UeZmC2FVGBAAjbANgAOkg9jy7urmsk34cA== 2024-09-15T03:16:07.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3UeZmC2FVGBAAjbANgAOkg9jy7urmsk34cA=="}' 2024-09-15T03:16:07.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c2d95d96-59e8-4a90-a9b8-958bac6836b8 -i td/ceph-helpers/0/new.json 2024-09-15T03:16:07.701 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:16:07.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:16:07.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3UeZmC2FVGBAAjbANgAOkg9jy7urmsk34cA== --osd-uuid c2d95d96-59e8-4a90-a9b8-958bac6836b8 2024-09-15T03:16:07.736 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:07.734+0000 7ff174122440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:07.747 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:07.746+0000 7ff174122440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:07.749 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:07.747+0000 7ff174122440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:07.749 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:07.747+0000 7ff174122440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:16:07.749 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:07.747+0000 7ff174122440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:16:07.749 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:07.747+0000 7ff174122440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:16:07.822 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:07.820+0000 7ff174122440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:16:13.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:16:13.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:16:13.485 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:16:13.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:16:13.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:16:13.972 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:16:13.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:16:13.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:16:13.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:16:13.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:16:13.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:16:13.994 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:13.993+0000 7f0587a4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:13.996 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:13.995+0000 7f0587a4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:13.997 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:13.996+0000 7f0587a4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:14.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:16:14.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:16:14.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:16:14.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:16:14.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:16:14.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:14.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:16:14.322 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:16:14.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:14.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:14.517 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:14.516+0000 7f0587a4f440 -1 Falling back to public interface 2024-09-15T03:16:14.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:15.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:15.701 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:16:15.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:15.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:16:15.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:15.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:16.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:16.999 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:16.997+0000 7f0587a4f440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:16:17.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:17.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:17.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:16:17.066 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:16:17.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:17.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:17.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:18.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:18.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:18.462 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:16:18.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:16:18.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:18.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:18.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:19.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:19.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:19.963 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:16:19.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:16:19.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:19.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:20.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:21.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:21.359 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:16:21.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:21.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:16:21.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:21.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:21.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:22.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:22.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:22.790 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:16:22.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:16:22.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:22.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:23.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:24.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:24.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:24.165 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:16:24.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:16:24.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:24.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:24.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:25.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:25.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:25.712 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:16:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:16:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:26.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:27.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:27.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:27.115 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:16:27.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:16:27.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:27.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:27.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:28.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:28.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:28.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:16:28.512 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:16:28.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:28.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:29.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:30.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:30.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:30.317 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:16:30.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:16:30.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:30.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:30.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:31.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:31.825 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:16:31.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:16:31.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:31.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:32.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:33.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:33.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:33.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:16:33.620 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:16:33.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:33.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:34.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:35.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:35.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:35.307 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:16:35.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:16:35.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:35.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:36.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:37.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:37.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:37.325 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:16:37.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:16:37.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:37.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:38.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:39.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:39.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:39.151 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:16:39.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:16:39.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:39.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:39.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:40.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:40.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:40.642 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:16:40.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:16:40.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:40.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:41.009 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/143003,v1:127.0.0.1:6803/143003] [v2:127.0.0.1:6804/143003,v1:127.0.0.1:6805/143003] exists,up c2d95d96-59e8-4a90-a9b8-958bac6836b8 2024-09-15T03:16:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:16:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:16:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:16:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-09-15T03:16:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:16:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:16:41.011 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:41.011 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:41.011 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:41.011 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.0.asok 2024-09-15T03:16:41.011 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-09-15T03:16:41.011 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.0.asok config get osd_max_backfills 2024-09-15T03:16:41.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-15T03:16:41.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-09-15T03:16:41.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-09-15T03:16:41.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-09-15T03:16:41.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:16:41.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:16:41.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:16:41.095 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:16:41.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:16:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=83f09197-4b7d-4bd1-8b2f-911082deb090 2024-09-15T03:16:41.098 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 83f09197-4b7d-4bd1-8b2f-911082deb090 2024-09-15T03:16:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 83f09197-4b7d-4bd1-8b2f-911082deb090' 2024-09-15T03:16:41.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:16:41.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZUeZmYcV/BhAAqCSqjpFr4pBf0hMU/qgD6A== 2024-09-15T03:16:41.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZUeZmYcV/BhAAqCSqjpFr4pBf0hMU/qgD6A=="}' 2024-09-15T03:16:41.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 83f09197-4b7d-4bd1-8b2f-911082deb090 -i td/ceph-helpers/1/new.json 2024-09-15T03:16:41.627 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:16:41.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:16:41.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQCZUeZmYcV/BhAAqCSqjpFr4pBf0hMU/qgD6A== --osd-uuid 83f09197-4b7d-4bd1-8b2f-911082deb090 2024-09-15T03:16:41.664 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:41.663+0000 7fe2f4008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:41.665 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:41.665+0000 7fe2f4008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:41.666 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:41.666+0000 7fe2f4008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:41.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:41.666+0000 7fe2f4008440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:16:41.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:41.666+0000 7fe2f4008440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:16:41.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:41.666+0000 7fe2f4008440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:16:41.709 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:41.709+0000 7fe2f4008440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:16:47.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:16:47.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:16:47.613 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:16:47.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:16:47.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:16:48.099 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:16:48.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:16:48.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-15T03:16:48.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:16:48.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:16:48.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:16:48.122 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:48.121+0000 7fb65182d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:48.124 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:48.123+0000 7fb65182d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:48.125 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:48.124+0000 7fb65182d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:48.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:16:48.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:16:48.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:16:48.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:16:48.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:16:48.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:48.451 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:16:48.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:16:48.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:48.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:48.654 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:48.653+0000 7fb65182d440 -1 Falling back to public interface 2024-09-15T03:16:48.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:49.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:49.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:49.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:16:49.802 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:16:49.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:49.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:50.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:51.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:51.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:51.157 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:16:51.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:16:51.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:51.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:51.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:16:51.250+0000 7fb65182d440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:16:51.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:52.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:52.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:52.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:16:52.594 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:16:52.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:52.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:53.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:54.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:54.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:54.099 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:16:54.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:16:54.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:54.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:54.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:55.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:55.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:55.499 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:16:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:16:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:55.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:56.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:56.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:56.943 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:16:56.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:16:56.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:56.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:57.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:58.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:58.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:58.416 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:16:58.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:16:58.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:58.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:58.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:59.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:59.813 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:16:59.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:59.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:16:59.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:59.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:00.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:01.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:01.251 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:17:01.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:01.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:17:01.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:01.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:01.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:02.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:02.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:02.724 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:17:02.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:17:02.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:02.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:03.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:04.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:04.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:04.254 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:17:04.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:17:04.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:04.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:05.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:06.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:06.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:06.577 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:17:06.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:17:06.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:06.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:07.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:08.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:08.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:08.354 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:17:08.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:17:08.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:08.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:09.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:10.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:10.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:10.156 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:17:10.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:17:10.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:10.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:10.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:11.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:11.951 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:17:11.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:11.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:17:11.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:11.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:12.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:13.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:13.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:13.853 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:17:13.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:17:13.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:13.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:14.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:15.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:15.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:15.769 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:17:15.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:17:15.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:15.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:16.142 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/144762,v1:127.0.0.1:6811/144762] [v2:127.0.0.1:6812/144762,v1:127.0.0.1:6813/144762] exists,up 83f09197-4b7d-4bd1-8b2f-911082deb090 2024-09-15T03:17:16.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:17:16.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:17:16.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:17:16.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-09-15T03:17:16.143 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:17:16.144 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:17:16.144 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:16.144 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:16.144 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:16.144 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.1.asok 2024-09-15T03:17:16.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-09-15T03:17:16.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.1.asok config get osd_max_backfills 2024-09-15T03:17:16.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-15T03:17:16.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:17:16.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:17:16.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:17:16.231 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:17:16.231 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:17:16.231 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:16.231 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:16.231 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:17:16.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:17:16.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:17:16.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:17:16.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:17:16.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:17:16.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-15T03:17:16.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:17:16.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f084c28e-e8a4-46b4-86a1-ead34e1c6e9f 2024-09-15T03:17:16.233 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 f084c28e-e8a4-46b4-86a1-ead34e1c6e9f 2024-09-15T03:17:16.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f084c28e-e8a4-46b4-86a1-ead34e1c6e9f' 2024-09-15T03:17:16.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:17:16.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8UeZmFQ9lDhAAvTdlL/RQNg2MletvzC6vKQ== 2024-09-15T03:17:16.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8UeZmFQ9lDhAAvTdlL/RQNg2MletvzC6vKQ=="}' 2024-09-15T03:17:16.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f084c28e-e8a4-46b4-86a1-ead34e1c6e9f -i td/ceph-helpers/2/new.json 2024-09-15T03:17:16.735 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:17:16.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-15T03:17:16.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8UeZmFQ9lDhAAvTdlL/RQNg2MletvzC6vKQ== --osd-uuid f084c28e-e8a4-46b4-86a1-ead34e1c6e9f 2024-09-15T03:17:16.772 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:16.770+0000 7f3683630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:16.774 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:16.772+0000 7f3683630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:16.775 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:16.773+0000 7f3683630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:16.775 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:16.774+0000 7f3683630440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:17:16.775 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:16.774+0000 7f3683630440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:17:16.775 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:16.774+0000 7f3683630440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:17:16.850 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:16.849+0000 7f3683630440 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-15T03:17:22.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-15T03:17:22.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:17:22.898 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:17:22.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:17:22.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:17:23.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:17:23.380 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:17:23.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:17:23.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:17:23.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:17:23.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:17:23.404 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:23.403+0000 7fd42c92a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:23.406 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:23.405+0000 7fd42c92a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:23.407 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:23.406+0000 7fd42c92a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:23.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:17:23.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:17:23.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:17:23.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:17:23.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:17:23.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:23.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:17:23.731 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:17:23.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:23.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:23.931 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:23.931+0000 7fd42c92a440 -1 Falling back to public interface 2024-09-15T03:17:24.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:25.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:25.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:25.088 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:17:25.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:17:25.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:25.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:25.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:26.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:26.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:26.445 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:17:26.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:17:26.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:26.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:26.459 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:26.458+0000 7fd42c92a440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:17:26.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:27.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:27.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:27.911 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:17:27.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:17:27.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:27.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:28.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:29.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:29.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:29.419 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:17:29.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:17:29.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:29.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:29.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:30.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:30.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:30.878 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:17:30.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:17:30.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:30.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:31.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:32.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:32.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:32.357 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:17:32.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:17:32.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:32.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:32.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:33.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:33.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:33.721 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:17:33.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:17:33.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:33.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:34.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:35.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:35.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:35.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:17:35.219 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:17:35.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:35.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:35.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:36.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:36.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:36.709 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:17:36.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:17:36.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:36.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:37.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:38.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:38.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:38.146 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:17:38.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:17:38.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:38.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:38.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:39.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:39.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:39.651 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:17:39.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:17:39.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:39.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:40.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:41.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:41.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:41.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:17:41.396 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:17:41.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:41.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:42.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:43.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:43.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:43.232 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:17:43.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:17:43.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:43.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:43.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:44.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:44.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:44.904 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:17:44.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:17:44.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:44.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:45.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:46.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:46.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:46.828 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:17:46.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:17:46.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:46.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:47.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:48.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:48.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:48.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:17:48.387 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:17:48.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:48.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:49.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:50.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:50.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:50.108 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:17:50.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:17:50.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:50.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:50.461 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/146519,v1:127.0.0.1:6819/146519] [v2:127.0.0.1:6820/146519,v1:127.0.0.1:6821/146519] exists,up f084c28e-e8a4-46b4-86a1-ead34e1c6e9f 2024-09-15T03:17:50.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:17:50.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:17:50.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:17:50.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-09-15T03:17:50.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:17:50.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:17:50.463 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:50.463 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:50.463 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:50.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-osd.2.asok 2024-09-15T03:17:50.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-09-15T03:17:50.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.77294/ceph-osd.2.asok config get osd_max_backfills 2024-09-15T03:17:50.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:50.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:50.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:50.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:50.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:50.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:50.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:50.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:50.652 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:50.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:50.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:50.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:50.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:50.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:50.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:50.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:50.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:50.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_set_config td/ceph-helpers 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:50.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:50.670 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:50.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:50.671 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:50.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:50.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:50.672 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:50.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:50.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:50.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:50.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:50.676 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:50.676 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:50.676 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:50.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:50.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:50.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:50.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:17:50.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:17:50.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:50.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:50.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:17:50.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:17:50.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:17:50.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:17:51.041 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:17:51.041 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:17:51.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:17:51.042 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:17:51.042 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.042 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:17:51.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:17:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.295 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:51.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:17:51.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:17:51.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:51.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:17:51.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:51.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:17:51.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get ms_crc_header 2024-09-15T03:17:51.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-09-15T03:17:51.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-09-15T03:17:51.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-15T03:17:51.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-15T03:17:51.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.519 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:51.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config set ms_crc_header false 2024-09-15T03:17:51.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:17:51.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-09-15T03:17:51.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:17:51.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:17:51.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:17:51.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get ms_crc_header 2024-09-15T03:17:51.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:51.666 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:51.667 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:51.667 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.667 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.667 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:51.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config set ms_crc_header true 2024-09-15T03:17:51.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:17:51.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-09-15T03:17:51.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:17:51.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:51.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:17:51.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get ms_crc_header 2024-09-15T03:17:51.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-09-15T03:17:51.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-09-15T03:17:51.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:51.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:51.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:51.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:51.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:51.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:51.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:51.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:51.918 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:51.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:51.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:51.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:51.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:51.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:51.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:51.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:51.922 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:51.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:51.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:51.925 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:51.925 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.925 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_setup td/ceph-helpers 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:51.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:51.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:51.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:51.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:51.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:51.928 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:51.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:51.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:51.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:51.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:51.930 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:51.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:51.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:51.931 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:51.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:51.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:51.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:51.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:51.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:51.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:51.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:17:51.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:17:51.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:51.939 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:51.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:51.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:51.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:51.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:51.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:51.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:51.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:51.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:51.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:51.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:51.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:51.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:51.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:51.943 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:51.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:51.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:51.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:51.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:51.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:51.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:51.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:17:51.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:17:51.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:51.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:51.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:51.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:51.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:51.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:51.953 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:51.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:51.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:51.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:51.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:51.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:51.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:51.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:51.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:51.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:51.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:51.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:51.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:51.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:17:51.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_teardown td/ceph-helpers 2024-09-15T03:17:51.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-09-15T03:17:51.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:51.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:51.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:51.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:51.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:51.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:51.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:51.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:51.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:51.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:51.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:51.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:51.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:51.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:51.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:51.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:51.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:51.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:51.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:17:51.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:17:51.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:17:51.972 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:17:51.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:17:51.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:17:51.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:17:51.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:51.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:51.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:51.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:51.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:51.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:51.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:51.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:51.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:51.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:51.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:51.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:51.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:51.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:51.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:51.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:51.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:51.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-09-15T03:17:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:17:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_background td/ceph-helpers 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: test_wait_background: local pids= 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147685"' 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147685' 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147687"' 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 1; exit 1' 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147687' 2024-09-15T03:17:51.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: test_wait_background: wait_background pids 2024-09-15T03:17:51.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 147685 147687' 2024-09-15T03:17:51.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-15T03:17:51.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-15T03:17:51.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147685 2024-09-15T03:17:51.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147688: /' 2024-09-15T03:17:51.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-15T03:17:51.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147690: /' 2024-09-15T03:17:52.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 1 2024-09-15T03:17:52.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: wait_background: return_code=1 2024-09-15T03:17:52.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-15T03:17:52.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147687 2024-09-15T03:17:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-15T03:17:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-15T03:17:53.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 1 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: test_wait_background: '[' 1 -ne 1 ']' 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147693"' 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147693' 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-15T03:17:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147695"' 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147695' 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_wait_background: wait_background pids 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 1; exit 0' 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 147693 147695' 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147693 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147696: /' 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-15T03:17:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147698: /' 2024-09-15T03:17:54.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-15T03:17:54.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-15T03:17:54.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147695 2024-09-15T03:17:55.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_wait_background: '[' 0 -ne 0 ']' 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_wait_background: '[' '!' -z '' ']' 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_clean td/ceph-helpers 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:17:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:17:55.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:17:55.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:17:55.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:17:55.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:17:55.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:17:55.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:17:55.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:17:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:17:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:17:55.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:17:55.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:17:55.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:17:55.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:17:55.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:17:55.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:55.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:55.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:17:55.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:17:55.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:17:55.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:17:56.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:17:56.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:56.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:56.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:17:56.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:17:56.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:17:56.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:17:56.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:17:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-15T03:17:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:17:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:17:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:17:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:17:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:17:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:17:56.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T03:17:56.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:17:56.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:17:56.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:17:56.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:17:56.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:56.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:17:56.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:17:56.708 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:17:56.709 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:17:56.709 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:56.709 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:56.709 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:17:56.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:17:56.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:17:56.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-09-15T03:17:56.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:17:56.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:17:56.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:17:56.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:17:56.787 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:17:56.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:17:56.789 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 deb3ead0-5e7a-4fc2-b509-edb1a3c0d7dc 2024-09-15T03:17:56.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=deb3ead0-5e7a-4fc2-b509-edb1a3c0d7dc 2024-09-15T03:17:56.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 deb3ead0-5e7a-4fc2-b509-edb1a3c0d7dc' 2024-09-15T03:17:56.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:17:56.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkUeZm9ie3LxAAJ2M9quXcs5FuRD1Qe7fV9Q== 2024-09-15T03:17:56.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkUeZm9ie3LxAAJ2M9quXcs5FuRD1Qe7fV9Q=="}' 2024-09-15T03:17:56.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new deb3ead0-5e7a-4fc2-b509-edb1a3c0d7dc -i td/ceph-helpers/0/new.json 2024-09-15T03:17:57.685 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:17:57.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:17:57.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkUeZm9ie3LxAAJ2M9quXcs5FuRD1Qe7fV9Q== --osd-uuid deb3ead0-5e7a-4fc2-b509-edb1a3c0d7dc 2024-09-15T03:17:57.719 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:57.717+0000 7f33ce830440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:57.731 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:57.729+0000 7f33ce830440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:57.732 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:57.730+0000 7f33ce830440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:57.732 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:57.731+0000 7f33ce830440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:17:57.732 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:57.731+0000 7f33ce830440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:17:57.732 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:57.731+0000 7f33ce830440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:17:57.772 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:17:57.770+0000 7f33ce830440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:18:03.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:18:03.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:18:03.366 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:18:03.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:18:03.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:18:03.561 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:18:03.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:18:03.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:18:03.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:18:03.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:18:03.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:18:03.583 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:03.581+0000 7fea7571d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:03.590 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:03.588+0000 7fea7571d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:03.591 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:03.589+0000 7fea7571d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:03.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:18:03.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:18:03.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:18:03.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:18:03.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:18:03.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:03.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:18:03.761 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:18:03.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:03.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:03.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:04.122 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:04.121+0000 7fea7571d440 -1 Falling back to public interface 2024-09-15T03:18:04.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:04.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:04.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:18:04.969 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:18:04.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:04.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:05.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:06.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:06.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:06.149 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:18:06.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:18:06.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:06.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:06.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:06.716 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:06.715+0000 7fea7571d440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:18:07.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:07.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:07.331 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:18:07.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:18:07.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:07.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:07.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:08.672 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:18:08.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:18:08.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:08.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:08.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:09.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:09.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:09.860 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:18:09.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:18:09.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:09.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:10.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:11.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:11.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:11.047 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:18:11.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:18:11.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:11.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:11.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:12.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:12.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:12.236 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:18:12.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:18:12.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:12.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:12.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:13.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:13.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:13.419 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:18:13.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:18:13.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:13.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:13.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:14.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:14.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:14.608 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:18:14.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:18:14.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:14.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:14.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:15.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:15.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:15.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:18:15.796 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:18:15.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:15.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:15.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:16.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:16.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:16.986 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:18:16.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:18:16.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:16.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:17.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:18.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:18.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:18.174 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:18:18.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:18:18.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:18.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:18.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:19.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:19.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:19.360 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:18:19.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:18:19.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:19.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:19.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:20.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:20.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:20.549 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:18:20.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:18:20.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:20.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:20.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:21.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:21.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:21.737 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:18:21.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:18:21.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:21.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:21.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:22.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:22.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:22.926 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:18:22.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:18:22.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:22.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:23.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:24.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:24.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:24.113 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:18:24.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:18:24.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:24.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:24.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:25.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:25.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:25.305 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:18:25.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:18:25.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:25.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:25.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:26.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:26.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:26.493 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:18:26.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:18:26.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:26.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:26.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:27.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:27.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:27.680 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:18:27.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:18:27.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:27.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:27.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:28.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:28.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:28.871 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-09-15T03:18:28.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-15T03:18:28.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:28.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:18:29.053 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/148636,v1:127.0.0.1:6801/148636] [v2:127.0.0.1:6802/148636,v1:127.0.0.1:6803/148636] exists,up deb3ead0-5e7a-4fc2-b509-edb1a3c0d7dc 2024-09-15T03:18:29.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:18:29.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:18:29.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:18:29.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:18:29.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:18:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:18:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:18:29.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:18:29.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:18:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:18:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-09-15T03:18:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:18:29.578 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:18:29.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:18:29.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:18:30.010 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:18:30.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:18:31.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:18:31.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-15T03:18:31.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:18:31.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:18:32.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-15T03:18:32.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:18:32.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:18:32.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:18:32.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:18:32.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:18:32.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:18:32.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:18:32.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:32.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:18:32.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=0 2024-09-15T03:18:32.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 0 2024-09-15T03:18:32.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-0' 2024-09-15T03:18:32.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:32.350 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-0 2024-09-15T03:18:32.350 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:32.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:18:32.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-0 2024-09-15T03:18:32.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:32.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=0 2024-09-15T03:18:32.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-15T03:18:32.353 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 0 2024-09-15T03:18:32.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:18:32.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 0 2024-09-15T03:18:32.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:18:32.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:32.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:32.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-15T03:18:32.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-15T03:18:33.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:18:33.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:33.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:34.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:18:34.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:18:34.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:18:34.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:18:34.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:18:34.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:18:34.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:18:34.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:18:34.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:18:34.587 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:18:34.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:34.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:35.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:18:35.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:18:35.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:18:35.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:18:35.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:18:35.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:18:35.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:18:35.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:18:35.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:18:35.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:18:35.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:18:35.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:18:35.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:18:35.194 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:18:35.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:18:35.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:18:35.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:35.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:35.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:18:35.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:18:35.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:18:35.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:18:35.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:18:35.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:18:35.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:18:35.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:18:35.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:18:36.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:18:36.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:18:36.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-15T03:18:36.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:18:36.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:18:36.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:18:36.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:18:36.673 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:18:36.673 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:18:36.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:18:36.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:18:36.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:18:36.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:18:37.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:18:37.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:18:37.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:37.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:37.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:18:37.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:18:37.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:18:37.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:18:37.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:18:37.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:18:37.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:18:37.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:18:37.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:18:37.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:18:37.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:18:37.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-15T03:18:37.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:18:37.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:18:38.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:18:38.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:18:38.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:18:38.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:18:38.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:18:38.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:18:38.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:18:38.352 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:18:38.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:18:38.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:18:38.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:38.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:18:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:18:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:18:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:18:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:18:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:18:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:18:39.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:18:39.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:18:39.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:18:39.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:18:39.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-15T03:18:39.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:18:39.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:18:39.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:18:40.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:18:40.272 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:18:40.272 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:40.272 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:40.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:18:40.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:18:41.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:18:41.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:18:41.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-15T03:18:41.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:18:41.668 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_fingerprint": "04a3c9d3-0d97-4349-8df0-329933f281a4", 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "timestamp": "2024-09-15T03:18:41.667661+0000", 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "tag": "", 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "health": { 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "HEALTH_WARN", 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "checks": { 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "PG_DEGRADED": { 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:18:41.669 INFO:tasks.workunit.client.0.smithi137.stdout: "summary": { 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: "count": 2 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: "detail": [ 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: "muted": false 2024-09-15T03:18:41.670 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "TOO_FEW_OSDS": { 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "summary": { 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "count": 2 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "detail": [], 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "muted": false 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "mutes": [] 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.671 INFO:tasks.workunit.client.0.smithi137.stdout: "monmap_first_committed": 1, 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "monmap_last_committed": 1, 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "monmap": { 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 1, 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "fsid": "145a3ccf-df94-4947-9f60-2f1acb4777f7", 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "modified": "2024-09-15T03:17:56.023019Z", 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "created": "2024-09-15T03:17:56.023019Z", 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "min_mon_release": 17, 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "election_strategy": 1, 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "disallowed_leaders: ": "", 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode": false, 2024-09-15T03:18:41.672 INFO:tasks.workunit.client.0.smithi137.stdout: "tiebreaker_mon": "", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "removed_ranks: ": "", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "features": { 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "persistent": [ 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "kraken", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "luminous", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "mimic", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap-prune", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "nautilus", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "octopus", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "pacific", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "elector-pinging", 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: "quincy" 2024-09-15T03:18:41.673 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "optional": [] 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "mons": [ 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "rank": 0, 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "a", 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addrs": { 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:7109", 2024-09-15T03:18:41.674 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 0 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:7109/0", 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: "priority": 0, 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 0, 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_location": "{}" 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.675 INFO:tasks.workunit.client.0.smithi137.stdout: "quorum": [ 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: 0 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap": { 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 13, 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "fsid": "145a3ccf-df94-4947-9f60-2f1acb4777f7", 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "created": "2024-09-15T03:17:56.935892+0000", 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "modified": "2024-09-15T03:18:31.720777+0000", 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "last_up_change": "2024-09-15T03:18:28.127899+0000", 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "last_in_change": "2024-09-15T03:17:57.549825+0000", 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "flags_num": 5799936, 2024-09-15T03:18:41.676 INFO:tasks.workunit.client.0.smithi137.stdout: "flags_set": [ 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "pglog_hardlimit", 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snapdirs", 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_deletes", 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "sortbitwise" 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_version": 3, 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_snapshot": "", 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_max": 1, 2024-09-15T03:18:41.677 INFO:tasks.workunit.client.0.smithi137.stdout: "max_osd": 1, 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "min_compat_client": "jewel", 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "require_osd_release": "quincy", 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "pools": [ 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_name": "rbd", 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "create_time": "2024-09-15T03:18:29.754049+0000", 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "flags": 8193, 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "type": 1, 2024-09-15T03:18:41.678 INFO:tasks.workunit.client.0.smithi137.stdout: "size": 2, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "min_size": 1, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_rule": 0, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "object_hash": 2, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_num": 4, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_placement_num": 4, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_placement_num_target": 4, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_num_target": 4, 2024-09-15T03:18:41.679 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_num_pending": 4, 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "last_pg_merge_meta": { 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "source_pgid": "0.0", 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "ready_epoch": 0, 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 0, 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 0, 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "source_version": "0'0", 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "target_version": "0'0" 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "13", 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "last_force_op_resend": "0", 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:18:41.680 INFO:tasks.workunit.client.0.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "auid": 0, 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_seq": 3, 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_epoch": 13, 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_snaps": [], 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "removed_snaps": "[]", 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "quota_max_bytes": 0, 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "quota_max_objects": 0, 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "tiers": [], 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "tier_of": -1, 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "read_tier": -1, 2024-09-15T03:18:41.681 INFO:tasks.workunit.client.0.smithi137.stdout: "write_tier": -1, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_mode": "none", 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "target_max_bytes": 0, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "target_max_objects": 0, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "erasure_code_profile": "", 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_params": { 2024-09-15T03:18:41.682 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "none" 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_period": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_count": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "use_gmt_hitset": true, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "grade_table": [], 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "stripe_width": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "expected_num_objects": 0, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "fast_read": false, 2024-09-15T03:18:41.683 INFO:tasks.workunit.client.0.smithi137.stdout: "options": {}, 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "application_metadata": { 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "rbd": {} 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "osds": [ 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "uuid": "deb3ead0-5e7a-4fc2-b509-edb1a3c0d7dc", 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "up": 1, 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "in": 1, 2024-09-15T03:18:41.684 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 1, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "primary_affinity": 1, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_begin": 0, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_end": 0, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "up_from": 5, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "up_thru": 7, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "down_at": 0, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "lost_at": 0, 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addrs": { 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6800", 2024-09-15T03:18:41.685 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6801", 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_addrs": { 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:18:41.686 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.687 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:18:41.688 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 148636 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addr": "127.0.0.1:6801/148636", 2024-09-15T03:18:41.689 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_addr": "127.0.0.1:6803/148636", 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_back_addr": "127.0.0.1:6807/148636", 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_front_addr": "127.0.0.1:6805/148636", 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "state": [ 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "exists", 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "up" 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_xinfo": [ 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "down_stamp": "0.000000", 2024-09-15T03:18:41.690 INFO:tasks.workunit.client.0.smithi137.stdout: "laggy_probability": 0, 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "laggy_interval": 0, 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "features": 4540138320759226367, 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "old_weight": 0, 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-15T03:18:27.783343+0000", 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "dead_epoch": 0 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_upmap": [], 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_upmap_items": [], 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_temp": [], 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "primary_temp": [], 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "blocklist": {}, 2024-09-15T03:18:41.691 INFO:tasks.workunit.client.0.smithi137.stdout: "range_blocklist": {}, 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "erasure_code_profiles": { 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "default": { 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "k": "2", 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "m": "2", 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "plugin": "jerasure", 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "technique": "reed_sol_van" 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "removed_snaps_queue": [ 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: "snaps": [ 2024-09-15T03:18:41.692 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: "begin": 2, 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: "length": 1 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: "new_removed_snaps": [ 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: "snaps": [ 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: "begin": 2, 2024-09-15T03:18:41.693 INFO:tasks.workunit.client.0.smithi137.stdout: "length": 1 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "new_purged_snaps": [], 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_node_flags": {}, 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "device_class_flags": {}, 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode": { 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:18:41.694 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_metadata": [ 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "arch": "x86_64", 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "back_addr": "[v2:127.0.0.1:6802/148636,v1:127.0.0.1:6803/148636]", 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "back_iface": "", 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs": "1", 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:18:41.695 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_release": "quincy", 2024-09-15T03:18:41.696 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "created_at": "2024-09-15T03:18:02.696236Z", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "default_device_class": "hdd", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XLAR", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "devices": "sda", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "distro": "centos", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "distro_version": "9", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "front_addr": "[v2:127.0.0.1:6800/148636,v1:127.0.0.1:6801/148636]", 2024-09-15T03:18:41.697 INFO:tasks.workunit.client.0.smithi137.stdout: "front_iface": "", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/148636,v1:127.0.0.1:6807/148636]", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/148636,v1:127.0.0.1:6805/148636]", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "hostname": "smithi137", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "journal_rotational": "1", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "mem_swap_kb": "0", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "mem_total_kb": "32487208", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "os": "Linux", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-15T03:18:41.698 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "osdspec_affinity": "", 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "rotational": "1" 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap_clean_epochs": { 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": { 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "per_pool": [] 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_epochs": [ 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:18:41.699 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 5 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap_first_committed": 1, 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap_last_committed": 13, 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: "crushmap": { 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: "devices": [ 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "osd.0", 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: "class": "hdd" 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.700 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: "types": [ 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 0, 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "osd" 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 1, 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "host" 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.701 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 2, 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "chassis" 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 3, 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "rack" 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 4, 2024-09-15T03:18:41.704 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "row" 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 5, 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "pdu" 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 6, 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "pod" 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 7, 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "room" 2024-09-15T03:18:41.705 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 8, 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "datacenter" 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 9, 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "zone" 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 10, 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "region" 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.706 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 11, 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "root" 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "buckets": [ 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -1, 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "default", 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 11, 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "root", 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:18:41.707 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -3, 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -2, 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "default~hdd", 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 11, 2024-09-15T03:18:41.708 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "root", 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -4, 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.709 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -3, 2024-09-15T03:18:41.710 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "smithi137", 2024-09-15T03:18:41.710 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 1, 2024-09-15T03:18:41.710 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "host", 2024-09-15T03:18:41.710 INFO:tasks.workunit.client.0.smithi137.stderr:report 3921204653 2024-09-15T03:18:41.710 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -4, 2024-09-15T03:18:41.711 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "smithi137~hdd", 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 1, 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "host", 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "rules": [ 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "rule_id": 0, 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "rule_name": "replicated_rule", 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "type": 1, 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "steps": [ 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "op": "take", 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "item": -1, 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi137.stdout: "item_name": "default" 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: "op": "choose_firstn", 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: "num": 0, 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "osd" 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: "op": "emit" 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.714 INFO:tasks.workunit.client.0.smithi137.stdout: "tunables": { 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_local_tries": 0, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_total_tries": 50, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "chooseleaf_stable": 1, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "straw_calc_version": 1, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "profile": "jewel", 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "optimal_tunables": 1, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "legacy_tunables": 0, 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "minimum_required_version": "jewel", 2024-09-15T03:18:41.715 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables": 1, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables2": 1, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v2_rules": 0, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables3": 1, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v3_rules": 0, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v4_buckets": 1, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables5": 1, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v5_rules": 0 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_args": {} 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "fsmap": { 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 1, 2024-09-15T03:18:41.716 INFO:tasks.workunit.client.0.smithi137.stdout: "default_fscid": -1, 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "compat": { 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "compat": {}, 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "ro_compat": {}, 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "incompat": { 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_1": "base v0.20", 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_8": "no anchor table", 2024-09-15T03:18:41.717 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_9": "file layout v2", 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_flags": { 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "enable_multiple": true, 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "ever_enabled_multiple": true 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "standbys": [], 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "filesystems": [] 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:18:41.718 INFO:tasks.workunit.client.0.smithi137.stdout: "auth": { 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "first_committed": 1, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "last_committed": 3, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_secrets": 1 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg": 4, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_active": 4, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_unknown": 0, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_osd": 1, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_sum": { 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 19, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 2, 2024-09-15T03:18:41.719 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 4, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 2, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 2, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 2, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 2, 2024-09-15T03:18:41.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-09-15T03:18:41.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.722 INFO:tasks.workunit.client.0.smithi137.stdout: "store_stats": { 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 0, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 0 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 3, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 3, 2024-09-15T03:18:41.723 INFO:tasks.workunit.client.0.smithi137.stdout: "up": 4, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": 4, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "num_store_stats": 0 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_sum": { 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "up_from": 0, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "seq": 0, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pgs": 4, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "num_osds": 1, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_osds": 1, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "kb": 104857600, 2024-09-15T03:18:41.724 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used": 297292, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_data": 116, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_omap": 1, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_meta": 297150, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_avail": 104560308, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "statfs": { 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 107374182400, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 107069755392, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 118784, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 24598, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:18:41.725 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 1704, 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 304281944 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_peers": [], 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "num_snap_trimming": 0, 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shards_repaired": 0, 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "op_queue_age_hist": { 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "histogram": [], 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "upper_bound": 1 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "perf_stat": { 2024-09-15T03:18:41.726 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ms": 0, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ms": 0, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ns": 0, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ns": 0 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "alerts": [], 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "network_ping_times": [] 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_sum_by_class": { 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "hdd": { 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "up_from": 0, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "seq": 0, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pgs": 4, 2024-09-15T03:18:41.727 INFO:tasks.workunit.client.0.smithi137.stdout: "num_osds": 1, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_osds": 1, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "kb": 104857600, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used": 297292, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_data": 116, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_omap": 1, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_meta": 297150, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_avail": 104560308, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "statfs": { 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 107374182400, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 107069755392, 2024-09-15T03:18:41.728 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 118784, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 24598, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 1704, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 304281944 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_peers": [], 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "num_snap_trimming": 0, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shards_repaired": 0, 2024-09-15T03:18:41.729 INFO:tasks.workunit.client.0.smithi137.stdout: "op_queue_age_hist": { 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "histogram": [], 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "upper_bound": 1 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "perf_stat": { 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ms": 0, 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ms": 0, 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ns": 0, 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ns": 0 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "alerts": [], 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: "network_ping_times": [] 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.730 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_stats": [ 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "poolid": 1, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg": 4, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 19, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 2, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 4, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 2, 2024-09-15T03:18:41.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 2, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 2, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 2, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-09-15T03:18:41.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-09-15T03:18:41.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "store_stats": { 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:18:41.734 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 4096, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 19, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 0, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 0 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 3, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 3, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "up": 4, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": 4, 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: "num_store_stats": 1 2024-09-15T03:18:41.735 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_stats": [ 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: "seq": 21474836482 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_by_state": [ 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: "num": 2 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.736 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized", 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "num": 2 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_by_osd": [ 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "num_primary_pg": 4, 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "num_acting_pg": 4, 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.737 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [ 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": {} 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "servicemap": { 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 1, 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "modified": "2024-09-15T03:17:56.936289+0000", 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "services": {} 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "paxos": { 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "first_committed": 1, 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "last_committed": 103, 2024-09-15T03:18:41.738 INFO:tasks.workunit.client.0.smithi137.stdout: "last_pn": 0, 2024-09-15T03:18:41.739 INFO:tasks.workunit.client.0.smithi137.stdout: "accepted_pn": 0 2024-09-15T03:18:41.739 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:18:41.739 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:18:41.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:41.743 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:18:41.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:18:41.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:18:41.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:18:41.745 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 b1c79e2b-9b7a-4ea9-a6fc-fa4ad2b401c3 2024-09-15T03:18:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b1c79e2b-9b7a-4ea9-a6fc-fa4ad2b401c3 2024-09-15T03:18:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b1c79e2b-9b7a-4ea9-a6fc-fa4ad2b401c3' 2024-09-15T03:18:41.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:18:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARUuZmLlWCKRAA7hlM8x3V3MUbPkvCcwiFeg== 2024-09-15T03:18:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARUuZmLlWCKRAA7hlM8x3V3MUbPkvCcwiFeg=="}' 2024-09-15T03:18:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b1c79e2b-9b7a-4ea9-a6fc-fa4ad2b401c3 -i td/ceph-helpers/1/new.json 2024-09-15T03:18:42.213 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:18:42.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:18:42.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARUuZmLlWCKRAA7hlM8x3V3MUbPkvCcwiFeg== --osd-uuid b1c79e2b-9b7a-4ea9-a6fc-fa4ad2b401c3 2024-09-15T03:18:42.248 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:42.246+0000 7fe661c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:42.249 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:42.248+0000 7fe661c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:42.250 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:42.249+0000 7fe661c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:42.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:42.249+0000 7fe661c0a440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:18:42.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:42.249+0000 7fe661c0a440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:18:42.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:42.249+0000 7fe661c0a440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:18:42.327 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:42.326+0000 7fe661c0a440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:18:47.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:18:47.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:18:47.981 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:18:47.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:18:47.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:18:48.465 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:18:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:18:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:18:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:18:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:18:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:18:48.487 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:48.486+0000 7fbe1b00e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:48.489 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:48.488+0000 7fbe1b00e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:48.490 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:48.489+0000 7fbe1b00e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:48.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:18:48.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:18:48.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:18:48.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:18:48.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:18:48.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:48.816 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:18:48.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:18:48.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:48.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:49.005 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:49.004+0000 7fbe1b00e440 -1 Falling back to public interface 2024-09-15T03:18:49.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:50.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:50.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:50.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:18:50.183 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:18:50.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:50.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:50.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:51.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:51.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:51.541 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:18:51.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:18:51.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:51.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:51.681 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:18:51.680+0000 7fbe1b00e440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:18:52.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:53.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:53.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:53.006 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:18:53.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:18:53.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:53.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:53.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:54.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:54.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:54.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:18:54.468 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:18:54.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:54.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:54.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:55.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:55.899 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:18:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:18:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:56.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:57.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:57.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:57.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:18:57.247 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:18:57.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:57.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:57.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:58.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:58.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:58.624 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:18:58.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:18:58.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:58.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:18:59.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:00.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:00.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:00.008 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:19:00.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:19:00.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:00.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:00.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:01.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:01.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:01.457 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:19:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:19:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:01.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:02.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:02.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:02.906 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:19:02.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:19:02.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:02.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:03.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:04.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:04.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:04.393 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:19:04.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:19:04.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:04.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:05.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:06.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:06.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:06.263 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:19:06.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:19:06.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:06.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:07.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:08.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:08.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:08.082 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:19:08.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:19:08.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:08.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:08.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:09.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:09.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:09.882 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:19:09.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:19:09.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:09.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:10.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:11.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:11.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:11.634 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:19:11.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:19:11.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:11.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:12.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:13.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:13.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:13.898 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:19:13.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:19:13.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:13.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:14.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:15.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:15.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:15.535 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:19:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:19:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:19:16.063 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/151221,v1:127.0.0.1:6811/151221] [v2:127.0.0.1:6812/151221,v1:127.0.0.1:6813/151221] exists,up b1c79e2b-9b7a-4ea9-a6fc-fa4ad2b401c3 2024-09-15T03:19:16.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:19:16.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:19:16.065 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:19:16.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:19:16.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:19:16.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:19:16.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:19:16.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:19:16.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:19:16.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:19:16.539 INFO:tasks.workunit.client.0.smithi137.stderr:1' 2024-09-15T03:19:16.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:19:16.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:19:16.539 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:19:16.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-15T03:19:16.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-15T03:19:16.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-15T03:19:16.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:19:16.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:19:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=73014444037 2024-09-15T03:19:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 73014444037 2024-09-15T03:19:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491 1-73014444037' 2024-09-15T03:19:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:19:16.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-15T03:19:16.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:19:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:19:16.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-15T03:19:16.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:19:16.762 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836491 2024-09-15T03:19:16.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-15T03:19:16.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-15T03:19:16.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:19:17.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-15T03:19:17.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:19:17.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-73014444037 2024-09-15T03:19:17.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:19:17.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:19:17.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-73014444037 2024-09-15T03:19:17.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:19:17.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=73014444037 2024-09-15T03:19:17.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 73014444037' 2024-09-15T03:19:17.118 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 73014444037 2024-09-15T03:19:17.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:19:17.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 73014444037 -lt 73014444037 2024-09-15T03:19:17.556 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:19:17.556 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:19:17.557 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:19:18.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:19:18.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:19:18.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:19:18.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:19:18.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:19:18.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:19:18.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:19:18.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:19:18.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:19:18.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:19:18.989 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:19:18.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:19:18.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:19:18.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:19:18.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:19:18.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:19:18.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:19:18.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:19:18.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:19:18.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:19:18.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:19:18.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:19:18.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:19:19.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:19:19.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:19.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:19.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:19:19.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:19:19.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:19:19.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:19:19.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1822: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1824: test_wait_for_health_ok: setup td/ceph-helpers 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:19:19.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:19:19.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:19:19.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:19:19.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:19:19.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:19:19.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:19:19.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:19:19.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:19:19.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:19:19.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:19:19.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:19:19.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:19:19.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:19:19.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:19:19.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:19:19.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:19:19.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:19.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:19.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:19:19.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:19:19.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:19:19.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:19:19.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:19:19.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:19.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:19.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:19:19.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:19:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:19:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:19:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:19:19.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1825: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-15T03:19:19.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:19:19.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:19:19.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:19:19.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:19:19.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:19:19.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:19:19.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-15T03:19:19.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:19:19.617 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:19.618 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:19:19.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:19:19.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:19:19.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:19.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:19:19.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:19:19.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:19:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1826: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-09-15T03:19:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:19:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:19:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:19:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:19:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:19:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:19:20.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:19:20.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:19:20.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:19:20.612 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:19:20.612 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:20.613 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:20.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:19:20.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:19:20.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-09-15T03:19:20.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1828: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-09-15T03:19:20.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:19:20.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:19:20.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:19:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:19:20.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:19:20.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36c57b73-0871-4074-a83f-e3290babf65e 2024-09-15T03:19:20.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 36c57b73-0871-4074-a83f-e3290babf65e' 2024-09-15T03:19:20.644 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 36c57b73-0871-4074-a83f-e3290babf65e 2024-09-15T03:19:20.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:19:20.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4UuZmqNMJJxAAnTN5e4V7Uh44Vfo42SwGlQ== 2024-09-15T03:19:20.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4UuZmqNMJJxAAnTN5e4V7Uh44Vfo42SwGlQ=="}' 2024-09-15T03:19:20.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36c57b73-0871-4074-a83f-e3290babf65e -i td/ceph-helpers/0/new.json 2024-09-15T03:19:20.973 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:19:20.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:19:20.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4UuZmqNMJJxAAnTN5e4V7Uh44Vfo42SwGlQ== --osd-uuid 36c57b73-0871-4074-a83f-e3290babf65e 2024-09-15T03:19:21.007 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:21.007+0000 7ff121f1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:21.018 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:21.018+0000 7ff121f1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:21.019 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:21.019+0000 7ff121f1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:21.020 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:21.019+0000 7ff121f1f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:19:21.020 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:21.019+0000 7ff121f1f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:19:21.020 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:21.019+0000 7ff121f1f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:19:21.058 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:21.058+0000 7ff121f1f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:19:26.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:19:26.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:19:26.423 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:19:26.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:19:26.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:19:26.905 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:19:26.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:19:26.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:19:26.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:19:26.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:19:26.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:19:26.928 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:26.927+0000 7fc734013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:26.930 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:26.929+0000 7fc734013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:26.930 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:26.930+0000 7fc734013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:27.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:19:27.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:19:27.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:19:27.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:19:27.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:19:27.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:27.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:19:27.256 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:19:27.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:27.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:27.447 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:27.447+0000 7fc734013440 -1 Falling back to public interface 2024-09-15T03:19:27.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:28.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:28.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:28.624 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:19:28.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:19:28.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:28.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:28.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:29.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:29.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:29.983 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:19:29.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:19:29.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:29.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:29.987 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:29.987+0000 7fc734013440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:19:30.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:31.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:31.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:31.355 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:19:31.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:19:31.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:31.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:31.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:32.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:32.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:32.818 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:19:32.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:19:32.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:32.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:33.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:34.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:34.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:34.206 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:19:34.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:19:34.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:34.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:34.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:35.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:35.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:35.689 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:19:35.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:19:35.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:35.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:36.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:37.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:37.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:37.072 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:19:37.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:19:37.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:37.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:37.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:38.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:38.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:38.452 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:19:38.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:19:38.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:38.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:38.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:39.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:39.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:39.938 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:19:39.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:19:39.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:39.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:40.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:41.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:41.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:41.305 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:19:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:19:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:42.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:42.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:42.755 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:19:42.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:19:42.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:42.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:43.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:44.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:44.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:44.232 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:19:44.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:19:44.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:44.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:44.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:45.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:45.997 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:19:45.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:45.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:19:45.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:45.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:46.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:47.553 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:19:47.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:19:47.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:47.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:48.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:49.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:49.360 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:19:49.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:49.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:19:49.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:49.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:50.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:51.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:51.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:51.204 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:19:51.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:19:51.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:51.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:52.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:19:53.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:19:53.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:19:53.007 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:19:53.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:19:53.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:19:53.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:19:53.373 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/153502,v1:127.0.0.1:6803/153502] [v2:127.0.0.1:6804/153502,v1:127.0.0.1:6805/153502] exists,up 36c57b73-0871-4074-a83f-e3290babf65e 2024-09-15T03:19:53.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:19:53.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:19:53.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:19:53.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-09-15T03:19:53.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:19:53.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:19:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:19:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:19:53.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:19:53.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:19:53.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2396ee01-ade3-4416-b582-3d42f95a4540 2024-09-15T03:19:53.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2396ee01-ade3-4416-b582-3d42f95a4540' 2024-09-15T03:19:53.382 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 2396ee01-ade3-4416-b582-3d42f95a4540 2024-09-15T03:19:53.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:19:53.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZUuZmKatNFxAAT3p3YppKpweMpI43PgPHEQ== 2024-09-15T03:19:53.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZUuZmKatNFxAAT3p3YppKpweMpI43PgPHEQ=="}' 2024-09-15T03:19:53.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2396ee01-ade3-4416-b582-3d42f95a4540 -i td/ceph-helpers/1/new.json 2024-09-15T03:19:53.871 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:19:53.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:19:53.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZUuZmKatNFxAAT3p3YppKpweMpI43PgPHEQ== --osd-uuid 2396ee01-ade3-4416-b582-3d42f95a4540 2024-09-15T03:19:53.907 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:53.905+0000 7fa6d804f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:53.908 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:53.907+0000 7fa6d804f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:53.910 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:53.908+0000 7fa6d804f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:19:53.910 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:53.908+0000 7fa6d804f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:19:53.910 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:53.908+0000 7fa6d804f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:19:53.910 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:53.908+0000 7fa6d804f440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:19:53.987 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:19:53.985+0000 7fa6d804f440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:19:59.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:19:59.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:19:59.892 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:19:59.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:19:59.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:20:00.374 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:20:00.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:20:00.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:20:00.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:20:00.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:20:00.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:20:00.396 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:00.394+0000 7f1ffd331440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:00.398 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:00.396+0000 7f1ffd331440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:00.399 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:00.397+0000 7f1ffd331440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:00.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:20:00.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:20:00.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:20:00.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:20:00.724 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:20:00.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:20:00.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:00.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:20:00.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:00.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:00.921 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:00.920+0000 7f1ffd331440 -1 Falling back to public interface 2024-09-15T03:20:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:02.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:02.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:02.078 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:20:02.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:20:02.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:02.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:02.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:03.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:03.428+0000 7f1ffd331440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:20:03.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:03.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:03.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:20:03.451 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:20:03.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:03.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:03.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:04.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:04.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:04.896 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:20:04.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:20:04.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:04.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:05.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:06.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:06.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:06.309 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:20:06.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:20:06.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:06.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:06.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:07.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:07.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:07.693 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:20:07.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:20:07.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:07.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:08.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:09.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:09.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:09.260 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:20:09.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:20:09.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:09.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:09.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:10.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:10.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:10.702 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:20:10.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:20:10.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:10.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:11.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:12.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:12.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:12.133 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:20:12.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:20:12.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:12.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:12.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:13.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:13.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:13.535 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:20:13.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:20:13.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:13.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:13.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:14.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:14.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:14.925 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:20:14.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:20:14.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:14.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:15.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:16.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:16.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:16.467 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:20:16.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:20:16.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:16.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:17.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:18.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:18.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:18.112 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:20:18.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:20:18.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:18.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:18.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:19.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:19.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:19.771 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:20:19.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:20:19.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:19.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:20.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:21.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:21.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:21.551 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:20:21.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:20:21.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:21.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:22.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:23.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:23.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:23.256 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:20:23.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:20:23.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:23.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:24.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:25.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:25.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:25.042 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:20:25.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:20:25.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:25.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:25.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:26.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:26.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:26.788 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:20:26.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:20:26.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:26.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:27.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:28.233 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:20:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:20:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:20:28.589 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/155258,v1:127.0.0.1:6811/155258] [v2:127.0.0.1:6812/155258,v1:127.0.0.1:6813/155258] exists,up 2396ee01-ade3-4416-b582-3d42f95a4540 2024-09-15T03:20:28.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:20:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:20:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:20:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:20:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:20:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-15T03:20:28.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:20:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd162ddc-3357-4a5b-9813-9775a616c27a 2024-09-15T03:20:28.596 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 fd162ddc-3357-4a5b-9813-9775a616c27a 2024-09-15T03:20:28.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fd162ddc-3357-4a5b-9813-9775a616c27a' 2024-09-15T03:20:28.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:20:28.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8UuZmnocmJBAArmWs4I7XmN2LCb2trbi2HA== 2024-09-15T03:20:28.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8UuZmnocmJBAArmWs4I7XmN2LCb2trbi2HA=="}' 2024-09-15T03:20:28.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd162ddc-3357-4a5b-9813-9775a616c27a -i td/ceph-helpers/2/new.json 2024-09-15T03:20:29.126 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:20:29.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-15T03:20:29.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8UuZmnocmJBAArmWs4I7XmN2LCb2trbi2HA== --osd-uuid fd162ddc-3357-4a5b-9813-9775a616c27a 2024-09-15T03:20:29.162 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:29.161+0000 7f8dd6d20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:29.163 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:29.163+0000 7f8dd6d20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:29.164 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:29.164+0000 7f8dd6d20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:29.165 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:29.164+0000 7f8dd6d20440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:20:29.165 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:29.164+0000 7f8dd6d20440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:20:29.165 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:29.164+0000 7f8dd6d20440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-15T03:20:29.241 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:29.241+0000 7f8dd6d20440 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-15T03:20:35.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-15T03:20:35.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:20:35.080 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:20:35.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:20:35.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:20:35.563 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:20:35.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:20:35.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:20:35.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:20:35.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:20:35.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:20:35.586 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:35.584+0000 7fc67e00b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:35.587 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:35.586+0000 7fc67e00b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:35.588 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:35.587+0000 7fc67e00b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:20:35.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:20:35.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:20:35.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:20:35.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:20:35.912 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:20:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:20:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:20:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:36.113 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:36.112+0000 7fc67e00b440 -1 Falling back to public interface 2024-09-15T03:20:36.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:37.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:37.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:37.259 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:20:37.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:20:37.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:37.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:37.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:38.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:38.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:38.618 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:20:38.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:20:38.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:38.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:38.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:20:38.701+0000 7fc67e00b440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:20:39.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:40.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:40.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:40.021 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:20:40.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:20:40.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:40.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:40.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:41.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:41.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:41.402 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:20:41.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:20:41.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:41.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:41.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:42.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:42.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:42.826 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:20:42.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:20:42.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:42.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:43.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:44.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:44.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:44.288 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:20:44.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:20:44.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:44.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:44.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:45.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:45.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:45.724 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:20:45.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:20:45.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:45.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:46.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:47.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:47.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:47.089 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:20:47.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:20:47.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:47.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:47.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:48.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:48.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:48.452 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:20:48.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:20:48.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:48.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:48.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:49.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:49.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:49.970 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:20:49.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:20:49.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:49.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:50.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:51.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:51.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:51.417 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:20:51.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:20:51.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:51.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:52.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:53.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:53.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:53.180 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:20:53.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:20:53.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:53.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:53.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:54.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:54.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:54.971 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:20:54.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:20:54.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:54.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:55.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:56.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:56.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:56.778 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:20:56.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:20:56.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:56.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:57.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:20:58.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:20:58.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:20:58.359 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:20:58.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:20:58.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:20:58.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:20:59.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:00.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:00.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:00.189 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:21:00.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:21:00.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:00.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:21:00.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:01.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:01.607 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:21:01.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:01.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:21:01.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:01.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:21:02.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:03.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:03.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:03.419 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:21:03.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:21:03.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:03.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:21:03.777 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/157038,v1:127.0.0.1:6819/157038] [v2:127.0.0.1:6820/157038,v1:127.0.0.1:6821/157038] exists,up fd162ddc-3357-4a5b-9813-9775a616c27a 2024-09-15T03:21:03.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:21:03.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:21:03.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:21:03.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-09-15T03:21:03.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:21:03.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:21:03.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:21:03.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:21:03.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:21:04.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:21:04.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: test_wait_for_health_ok: ceph osd down 0 2024-09-15T03:21:04.544 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already down. 2024-09-15T03:21:04.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: TIMEOUT=1 2024-09-15T03:21:04.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: wait_for_health_ok 2024-09-15T03:21:04.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-15T03:21:04.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-09-15T03:21:04.556 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 1 .1 2024-09-15T03:21:04.556 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:21:04.557 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:21:04.557 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:21:04.557 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:21:04.557 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:21:04.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-15T03:21:04.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-09-15T03:21:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-09-15T03:21:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-15T03:21:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-15T03:21:05.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 0 >= 4 )) 2024-09-15T03:21:05.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.1 2024-09-15T03:21:05.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-15T03:21:05.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-15T03:21:05.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-15T03:21:05.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 1 >= 4 )) 2024-09-15T03:21:05.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.2 2024-09-15T03:21:05.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-15T03:21:05.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-15T03:21:05.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-15T03:21:06.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 2 >= 4 )) 2024-09-15T03:21:06.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.4 2024-09-15T03:21:06.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-15T03:21:06.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-15T03:21:06.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-15T03:21:07.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 3 >= 4 )) 2024-09-15T03:21:07.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.3 2024-09-15T03:21:07.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-15T03:21:07.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-15T03:21:07.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-15T03:21:07.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 4 >= 4 )) 2024-09-15T03:21:07.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1803: wait_for_health: ceph health detail 2024-09-15T03:21:08.441 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-09-15T03:21:08.441 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] OSD_DOWN: 3 osds down 2024-09-15T03:21:08.441 INFO:tasks.workunit.client.0.smithi137.stdout: osd.0 (root=default,host=smithi137) is down 2024-09-15T03:21:08.441 INFO:tasks.workunit.client.0.smithi137.stdout: osd.1 (root=default,host=smithi137) is down 2024-09-15T03:21:08.442 INFO:tasks.workunit.client.0.smithi137.stdout: osd.2 (root=default,host=smithi137) is down 2024-09-15T03:21:08.442 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-09-15T03:21:08.442 INFO:tasks.workunit.client.0.smithi137.stdout: host smithi137 (root=default) (3 osds) is down 2024-09-15T03:21:08.442 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-09-15T03:21:08.442 INFO:tasks.workunit.client.0.smithi137.stdout: root default (3 osds) is down 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1804: wait_for_health: return 1 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: return 1 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:21:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:21:08.456 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:21:08.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:21:08.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:21:08.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:21:08.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:21:08.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:21:08.458 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:21:08.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:21:08.459 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-15T03:21:08.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:21:08.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:21:08.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:21:08.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:21:08.477 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:08.476+0000 7fa086c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:08.500 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:08.499+0000 7fa086c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:08.501 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:08.501+0000 7fa086c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:08.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:21:08.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:21:08.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:21:08.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:21:08.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:21:08.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:08.844 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:21:08.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:21:08.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:08.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:09.019 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:09.018+0000 7fa086c09440 -1 Falling back to public interface 2024-09-15T03:21:09.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:10.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:10.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:10.224 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:21:10.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:21:10.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:10.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:10.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:11.587 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:21:11.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:21:11.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:11.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:11.783 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:11.782+0000 7fa086c09440 -1 osd.0 14 log_to_monitors true 2024-09-15T03:21:11.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:12.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:12.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:12.954 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:21:12.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:21:12.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:12.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:13.309 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,14) [v2:127.0.0.1:6802/158200,v1:127.0.0.1:6803/158200] [v2:127.0.0.1:6804/158200,v1:127.0.0.1:6805/158200] exists,up 36c57b73-0871-4074-a83f-e3290babf65e 2024-09-15T03:21:13.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:21:13.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:21:13.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:21:13.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-09-15T03:21:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-15T03:21:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:21:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:21:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:21:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:21:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:21:13.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:21:13.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:21:13.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:21:13.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:21:13.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:21:13.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:21:13.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:21:13.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:21:13.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:21:13.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:21:13.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:21:13.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:21:13.320 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:13.320 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:13.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:13.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:13.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:21:13.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:21:13.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:21:13.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:21:13.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:21:13.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:21:13.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:21:13.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:21:13.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:21:13.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:21:13.322 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:21:13.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:21:13.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:21:13.325 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-09-15T03:21:13.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:21:13.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:21:13.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:21:13.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:21:13.333 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:13.332+0000 7f2f5142c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:13.357 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:13.357+0000 7f2f5142c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:13.358 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:13.358+0000 7f2f5142c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:21:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:21:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:21:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:21:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:21:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:13.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:21:13.695 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:21:13.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:13.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:21:13.876 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:13.876+0000 7f2f5142c440 -1 Falling back to public interface 2024-09-15T03:21:14.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:15.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:15.046 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:21:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:21:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:21:15.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:16.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:16.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:16.399 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:21:16.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:21:16.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:16.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:21:16.689 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:16.688+0000 7f2f5142c440 -1 osd.1 16 log_to_monitors true 2024-09-15T03:21:16.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:17.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:17.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:17.759 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:21:17.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:21:17.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:17.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:21:18.112 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6810/158778,v1:127.0.0.1:6811/158778] [v2:127.0.0.1:6812/158778,v1:127.0.0.1:6813/158778] exists,up 2396ee01-ade3-4416-b582-3d42f95a4540 2024-09-15T03:21:18.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:21:18.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:21:18.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:21:18.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-09-15T03:21:18.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:21:18.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:21:18.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:21:18.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:21:18.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:21:18.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-09-15T03:21:18.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:21:18.119 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:21:18.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:21:18.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-09-15T03:21:18.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:21:18.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:21:18.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:21:18.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:21:18.137 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:18.137+0000 7f3bd8c17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:18.153 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:18.153+0000 7f3bd8c17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:18.154 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:18.154+0000 7f3bd8c17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:21:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:21:18.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:21:18.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:21:18.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:21:18.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:18.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:21:18.481 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:21:18.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:18.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:21:18.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:18.667+0000 7f3bd8c17440 -1 Falling back to public interface 2024-09-15T03:21:18.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:19.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:19.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:19.834 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:21:19.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:21:19.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:19.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:21:20.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:21.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:21.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:21.194 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:21:21.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:21:21.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:21.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:21:21.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:21.937 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:21.937+0000 7f3bd8c17440 -1 osd.2 15 log_to_monitors true 2024-09-15T03:21:22.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:22.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:22.565 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:21:22.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:21:22.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:22.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:21:22.937 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6818/159356,v1:127.0.0.1:6819/159356] [v2:127.0.0.1:6820/159356,v1:127.0.0.1:6821/159356] exists,up fd162ddc-3357-4a5b-9813-9775a616c27a 2024-09-15T03:21:22.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: test_wait_for_health_ok: wait_for_health_ok 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 300 .1 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:21:22.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:21:23.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-15T03:21:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-09-15T03:21:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-09-15T03:21:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-15T03:21:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-15T03:21:23.586 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_OK 2024-09-15T03:21:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: test_wait_for_health_ok: teardown td/ceph-helpers 2024-09-15T03:21:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:21:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:21:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:21:23.587 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:21:23.587 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:21:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:21:23.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:21:23.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:21:23.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:21:23.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:21:23.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:21:23.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:21:23.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:21:23.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:21:23.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:21:23.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:21:23.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:21:23.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:21:23.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:21:23.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:21:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:21:23.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:21:23.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:21:23.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:23.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:23.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_osd td/ceph-helpers 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:21:23.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:21:23.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:21:23.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:21:23.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:21:23.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:21:23.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:21:23.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:21:23.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:21:23.718 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:21:23.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:21:23.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:21:23.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:21:23.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:21:23.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:21:23.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:21:23.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:21:23.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:21:23.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:23.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:23.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:21:23.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:21:23.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:21:23.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:21:23.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:21:23.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:23.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:23.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:21:23.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:21:23.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:24.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:21:24.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:21:24.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:21:24.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:21:24.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:21:24.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:21:24.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:21:24.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:21:24.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:21:24.329 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:21:24.329 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:21:24.329 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:24.329 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:24.330 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:21:24.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:21:24.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:21:24.404 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:24.405 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:21:24.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:21:24.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:21:24.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2024-09-15T03:21:24.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:21:24.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:21:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:21:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:21:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:21:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:21:25.314 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:21:25.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:21:25.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2024-09-15T03:21:25.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:21:25.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:21:25.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:21:25.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:21:25.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:21:25.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:21:25.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:21:25.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:21:25.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90571ca5-7ca0-47e9-aeae-087ce656fefe 2024-09-15T03:21:25.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 90571ca5-7ca0-47e9-aeae-087ce656fefe' 2024-09-15T03:21:25.346 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 90571ca5-7ca0-47e9-aeae-087ce656fefe 2024-09-15T03:21:25.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:21:25.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1UuZm2PBFFRAATQIa67MAKKkSTsUEDm914A== 2024-09-15T03:21:25.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1UuZm2PBFFRAATQIa67MAKKkSTsUEDm914A=="}' 2024-09-15T03:21:25.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90571ca5-7ca0-47e9-aeae-087ce656fefe -i td/ceph-helpers/0/new.json 2024-09-15T03:21:25.666 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:21:25.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:21:25.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1UuZm2PBFFRAATQIa67MAKKkSTsUEDm914A== --osd-uuid 90571ca5-7ca0-47e9-aeae-087ce656fefe 2024-09-15T03:21:25.700 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:25.699+0000 7f1f32248440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:25.711 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:25.711+0000 7f1f32248440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:25.712+0000 7f1f32248440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:25.712+0000 7f1f32248440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:21:25.713 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:25.712+0000 7f1f32248440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:21:25.713 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:25.712+0000 7f1f32248440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:21:25.752 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:25.751+0000 7f1f32248440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:21:31.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:21:31.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:21:31.058 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:21:31.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:21:31.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:21:31.550 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:21:31.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:21:31.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:21:31.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:21:31.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:21:31.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:21:31.572 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:31.570+0000 7ff9bca08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:31.574 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:31.573+0000 7ff9bca08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:31.575 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:31.574+0000 7ff9bca08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:31.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:21:31.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:21:31.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:21:31.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:21:31.941 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:21:31.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:21:31.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:31.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:21:31.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:31.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:32.108 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:32.107+0000 7ff9bca08440 -1 Falling back to public interface 2024-09-15T03:21:32.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:33.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:33.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:33.298 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:21:33.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:21:33.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:33.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:33.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:34.571 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:34.569+0000 7ff9bca08440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:21:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:21:34.655 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:21:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:35.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:36.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:36.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:36.148 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:21:36.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:21:36.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:36.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:36.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:37.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:37.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:37.576 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:21:37.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:21:37.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:37.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:37.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:38.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:38.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:38.983 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:21:38.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:21:38.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:38.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:39.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:40.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:40.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:40.388 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:21:40.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:21:40.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:40.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:40.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:41.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:41.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:41.745 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:21:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:21:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:41.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:42.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:43.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:43.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:43.251 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:21:43.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:21:43.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:43.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:43.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:44.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:44.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:44.638 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:21:44.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:21:44.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:44.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:45.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:46.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:46.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:46.036 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:21:46.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:21:46.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:46.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:46.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:47.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:47.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:47.717 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:21:47.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:21:47.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:47.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:48.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:49.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:49.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:49.491 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:21:49.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:21:49.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:49.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:50.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:51.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:51.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:51.225 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:21:51.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:21:51.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:51.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:51.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:52.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:52.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:52.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:21:52.821 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:21:52.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:52.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:54.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:54.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:54.396 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:21:54.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:21:54.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:54.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:55.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:21:56.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:21:56.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:21:56.068 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:21:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:21:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:21:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:21:56.713 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/161170,v1:127.0.0.1:6803/161170] [v2:127.0.0.1:6804/161170,v1:127.0.0.1:6805/161170] exists,up 90571ca5-7ca0-47e9-aeae-087ce656fefe 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:21:56.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:21:56.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:21:56.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:21:56.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:21:56.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:21:56.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:21:56.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:21:56.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:21:56.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:21:56.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:21:56.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:21:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:21:56.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:21:56.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:21:56.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=212564b9-4165-45df-a570-2e4f5e4f3fd8 2024-09-15T03:21:56.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 212564b9-4165-45df-a570-2e4f5e4f3fd8' 2024-09-15T03:21:56.723 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 212564b9-4165-45df-a570-2e4f5e4f3fd8 2024-09-15T03:21:56.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:21:56.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUUuZmP+2JKxAAg1B87LExpQAxBv1/4gbEQA== 2024-09-15T03:21:56.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUUuZmP+2JKxAAg1B87LExpQAxBv1/4gbEQA=="}' 2024-09-15T03:21:56.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 212564b9-4165-45df-a570-2e4f5e4f3fd8 -i td/ceph-helpers/1/new.json 2024-09-15T03:21:57.214 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:21:57.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:21:57.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUUuZmP+2JKxAAg1B87LExpQAxBv1/4gbEQA== --osd-uuid 212564b9-4165-45df-a570-2e4f5e4f3fd8 2024-09-15T03:21:57.250 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:57.248+0000 7f3bf8038440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:57.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:57.250+0000 7f3bf8038440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:57.252 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:57.251+0000 7f3bf8038440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:21:57.253 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:57.251+0000 7f3bf8038440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:21:57.253 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:57.251+0000 7f3bf8038440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:21:57.253 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:57.251+0000 7f3bf8038440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:21:57.329 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:21:57.328+0000 7f3bf8038440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:22:03.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:22:03.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:22:03.398 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:22:03.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:22:03.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:22:03.892 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:22:03.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:22:03.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:22:03.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:22:03.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:22:03.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:22:03.914 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:03.913+0000 7ff8e383b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:03.915 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:03.915+0000 7ff8e383b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:03.916 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:03.916+0000 7ff8e383b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:04.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:22:04.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:22:04.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:22:04.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:22:04.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:22:04.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:04.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:22:04.288 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:22:04.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:04.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:04.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:04.430+0000 7ff8e383b440 -1 Falling back to public interface 2024-09-15T03:22:04.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:05.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:05.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:05.641 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:22:05.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:22:05.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:05.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:05.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:06.942 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:06.941+0000 7ff8e383b440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:22:06.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:06.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:06.995 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:22:06.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:22:06.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:06.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:07.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:08.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:08.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:08.417 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:22:08.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:22:08.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:08.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:08.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:09.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:09.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:09.889 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:22:09.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:22:09.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:09.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:10.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:11.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:11.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:11.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:22:11.304 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:22:11.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:11.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:11.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:12.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:12.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:12.751 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:22:12.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:22:12.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:12.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:13.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:14.157 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:22:14.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:22:14.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:14.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:14.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:15.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:15.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:15.554 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:22:15.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:22:15.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:15.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:15.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:16.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:16.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:16.965 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:22:16.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:22:16.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:16.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:17.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:18.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:18.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:18.329 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:22:18.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:22:18.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:18.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:18.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:19.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:19.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:19.800 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:22:19.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:22:19.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:19.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:20.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:21.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:21.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:21.634 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:22:21.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:22:21.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:21.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:22.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:23.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:23.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:23.394 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:22:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:22:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:24.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:25.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:25.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:25.234 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:22:25.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:22:25.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:25.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:25.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:26.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:26.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:26.900 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:22:26.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:22:26.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:26.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:27.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:28.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:28.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:28.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:22:28.443 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:22:28.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:28.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:29.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:30.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:30.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:30.025 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:22:30.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:22:30.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:30.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:30.376 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/162899,v1:127.0.0.1:6811/162899] [v2:127.0.0.1:6812/162899,v1:127.0.0.1:6813/162899] exists,up 212564b9-4165-45df-a570-2e4f5e4f3fd8 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:30.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:22:30.379 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:22:30.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:30.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:30.728 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/161170,v1:127.0.0.1:6803/161170] [v2:127.0.0.1:6804/161170,v1:127.0.0.1:6805/161170] exists,up 90571ca5-7ca0-47e9-aeae-087ce656fefe 2024-09-15T03:22:30.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:22:30.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:22:30.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:22:30.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2024-09-15T03:22:30.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:22:30.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:22:30.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:22:30.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:22:30.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:30.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:22:30.730 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:22:30.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:30.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:22:31.075 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/162899,v1:127.0.0.1:6811/162899] [v2:127.0.0.1:6812/162899,v1:127.0.0.1:6813/162899] exists,up 212564b9-4165-45df-a570-2e4f5e4f3fd8 2024-09-15T03:22:31.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:22:31.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:22:31.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:22:31.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-15T03:22:31.078 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:22:31.078 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:22:31.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:22:31.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:22:31.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:22:31.383 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:22:31.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:31.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-15T03:22:31.783 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 12 last_clean_interval [0,0) [v2:127.0.0.1:6802/161170,v1:127.0.0.1:6803/161170] [v2:127.0.0.1:6804/161170,v1:127.0.0.1:6805/161170] exists 90571ca5-7ca0-47e9-aeae-087ce656fefe 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-15T03:22:31.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:22:31.786 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:22:31.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:31.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:32.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:33.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:33.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-15T03:22:33.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2024-09-15T03:22:33.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2024-09-15T03:22:33.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:22:33.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:22:33.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:22:33.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:22:33.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:22:33.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:22:33.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:22:33.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:22:33.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:22:33.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:22:33.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:22:33.244 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:22:33.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:22:33.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:22:33.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:22:33.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:33.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:22:33.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:22:33.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:33.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:22:33.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:22:33.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:22:33.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:22:33.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:22:33.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:22:33.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:22:33.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_peered td/ceph-helpers 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: test_wait_for_peered: local dir=td/ceph-helpers 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1736: test_wait_for_peered: setup td/ceph-helpers 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:22:33.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:22:33.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:22:33.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:22:33.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:22:33.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:22:33.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:22:33.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:22:33.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:22:33.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:22:33.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:22:33.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:22:33.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:22:33.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:33.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:22:33.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:22:33.266 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:33.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:22:33.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:22:33.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:22:33.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:22:33.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:22:33.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:22:33.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:22:33.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:22:33.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:22:33.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:22:33.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:22:33.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:22:33.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:22:33.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:22:33.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:22:33.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:22:33.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-15T03:22:33.671 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:22:33.672 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:22:33.672 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:22:33.672 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:22:33.672 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:22:33.672 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:22:33.672 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:22:33.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T03:22:33.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:22:33.932 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:22:33.933 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:22:33.933 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:22:33.933 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:22:33.933 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:22:33.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:22:33.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:22:34.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:22:34.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:22:34.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:22:34.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:22:34.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:22:34.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: test_wait_for_peered: run_osd td/ceph-helpers 0 2024-09-15T03:22:34.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:22:34.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:22:34.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:22:34.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:22:34.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:22:34.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6527c780-3561-4de7-a52a-5f7f023ba725 2024-09-15T03:22:34.095 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 6527c780-3561-4de7-a52a-5f7f023ba725 2024-09-15T03:22:34.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6527c780-3561-4de7-a52a-5f7f023ba725' 2024-09-15T03:22:34.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:22:34.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6UuZmSZtRBhAAzcVjEk9luM2YSob6Lvn5rA== 2024-09-15T03:22:34.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6UuZmSZtRBhAAzcVjEk9luM2YSob6Lvn5rA=="}' 2024-09-15T03:22:34.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6527c780-3561-4de7-a52a-5f7f023ba725 -i td/ceph-helpers/0/new.json 2024-09-15T03:22:34.975 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:22:34.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:22:34.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6UuZmSZtRBhAAzcVjEk9luM2YSob6Lvn5rA== --osd-uuid 6527c780-3561-4de7-a52a-5f7f023ba725 2024-09-15T03:22:35.010 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:35.008+0000 7fec48c11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:35.013 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:35.012+0000 7fec48c11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:35.014 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:35.013+0000 7fec48c11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:35.015 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:35.013+0000 7fec48c11440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:22:35.015 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:35.013+0000 7fec48c11440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:22:35.015 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:35.013+0000 7fec48c11440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:22:35.046 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:35.044+0000 7fec48c11440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:22:40.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:22:40.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:22:40.326 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:22:40.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:22:40.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:22:40.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:22:40.522 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:22:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:22:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:22:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:22:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:22:40.544 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:40.542+0000 7ff2ac782440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:40.556 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:40.554+0000 7ff2ac782440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:40.557 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:40.555+0000 7ff2ac782440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:22:40.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:22:40.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:22:40.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:22:40.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:22:40.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:22:40.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:40.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:22:40.728 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:22:40.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:40.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:40.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:41.085 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:41.084+0000 7ff2ac782440 -1 Falling back to public interface 2024-09-15T03:22:41.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:41.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:41.929 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:22:41.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:22:41.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:41.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:42.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:43.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:43.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:43.113 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:22:43.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:22:43.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:43.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:43.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:43.641 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:22:43.639+0000 7ff2ac782440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:22:44.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:44.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:44.302 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:22:44.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:22:44.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:44.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:44.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:45.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:45.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:45.587 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:22:45.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:22:45.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:45.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:45.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:46.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:46.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:46.772 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:22:46.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:22:46.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:46.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:46.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:47.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:47.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:47.959 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:22:47.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:22:47.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:47.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:48.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:49.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:49.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:49.148 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:22:49.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:22:49.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:49.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:49.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:50.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:50.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:50.337 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:22:50.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:22:50.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:50.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:50.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:51.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:51.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:51.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:22:51.529 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:22:51.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:51.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:51.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:52.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:52.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:52.717 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:22:52.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:22:52.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:52.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:52.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:53.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:53.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:53.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:22:53.908 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:22:53.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:53.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:54.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:55.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:55.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:55.095 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:22:55.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:22:55.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:55.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:55.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:56.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:56.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:56.284 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:22:56.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:22:56.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:56.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:56.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:57.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:57.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:57.466 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:22:57.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:22:57.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:57.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:57.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:58.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:58.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:58.645 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:22:58.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:22:58.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:58.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:22:58.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:22:59.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:22:59.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:22:59.825 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:22:59.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:22:59.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:22:59.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:23:00.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:01.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:01.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:01.009 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:23:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:23:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:23:01.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:02.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:02.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:02.197 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:23:02.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:23:02.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:02.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:23:02.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:03.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:03.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:03.385 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:23:03.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:23:03.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:03.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:23:03.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:04.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:04.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:04.587 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:23:04.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:23:04.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:04.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:23:04.765 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/164886,v1:127.0.0.1:6801/164886] [v2:127.0.0.1:6802/164886,v1:127.0.0.1:6803/164886] exists,up 6527c780-3561-4de7-a52a-5f7f023ba725 2024-09-15T03:23:04.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1739: test_wait_for_peered: run_mgr td/ceph-helpers x 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:23:04.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:23:05.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:23:05.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:23:05.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:23:05.193 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:23:05.193 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:05.193 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:05.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:23:05.194 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:23:05.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:23:05.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: test_wait_for_peered: create_rbd_pool 2024-09-15T03:23:05.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:23:05.426 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:23:05.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:23:05.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:23:05.761 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:23:05.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:23:06.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:23:07.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: wait_for_clean 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:23:07.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:23:07.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-15T03:23:07.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:23:07.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:23:07.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:23:07.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:23:07.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:23:07.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:23:07.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:23:07.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:23:07.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:23:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=0 2024-09-15T03:23:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 0 2024-09-15T03:23:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-0' 2024-09-15T03:23:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:23:07.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-0 2024-09-15T03:23:07.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:23:07.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:23:07.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-0 2024-09-15T03:23:07.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:23:07.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=0 2024-09-15T03:23:07.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-15T03:23:07.963 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 0 2024-09-15T03:23:07.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:23:08.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 0 2024-09-15T03:23:08.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:23:08.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:08.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:08.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-15T03:23:08.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-15T03:23:09.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:23:09.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:09.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:23:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:23:09.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:23:09.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:23:09.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:23:09.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:23:09.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:23:09.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:23:10.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:23:10.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:23:10.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:10.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:10.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:23:10.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:23:10.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:23:10.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:23:10.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:23:10.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:23:10.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:23:10.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:23:10.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:23:10.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:23:10.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:23:10.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:23:10.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:23:10.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:23:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:23:11.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:23:11.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:11.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:11.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:23:11.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:23:12.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:23:12.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:23:12.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-15T03:23:12.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:23:12.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:23:12.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:23:12.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:23:12.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:23:12.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:12.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:13.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:23:13.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:23:13.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:23:13.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:23:13.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:23:13.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:23:13.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:23:13.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:23:13.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:23:13.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:23:13.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:23:13.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-15T03:23:13.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:23:13.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:23:13.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:23:13.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:23:13.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:23:13.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:23:13.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:23:13.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:23:13.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:23:13.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:23:14.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:23:14.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:23:14.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:14.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:23:14.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:23:15.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:23:15.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:23:15.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-15T03:23:15.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:23:15.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-15T03:23:15.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:23:15.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:23:15.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:23:15.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:23:15.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:23:15.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:23:15.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:23:15.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:23:15.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:23:15.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:23:15.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:15.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:16.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-15T03:23:16.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:23:16.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:23:16.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:23:16.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:23:16.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:23:16.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:23:16.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:23:16.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:23:16.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:23:16.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:23:16.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-15T03:23:16.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:23:17.241 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-09-15T03:23:17.241 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_fingerprint": "25186b05-25e9-4ec3-b5bc-af667d27ee25", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "timestamp": "2024-09-15T03:23:17.240311+0000", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "tag": "", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "health": { 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "HEALTH_WARN", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "checks": { 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "PG_DEGRADED": { 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "summary": { 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-15T03:23:17.242 INFO:tasks.workunit.client.0.smithi137.stdout: "count": 2 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: "detail": [ 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: "muted": false 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.243 INFO:tasks.workunit.client.0.smithi137.stdout: "TOO_FEW_OSDS": { 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "summary": { 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "count": 2 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "detail": [], 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "muted": false 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "mutes": [] 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "monmap_first_committed": 1, 2024-09-15T03:23:17.244 INFO:tasks.workunit.client.0.smithi137.stdout: "monmap_last_committed": 1, 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "monmap": { 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 1, 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "fsid": "145a3ccf-df94-4947-9f60-2f1acb4777f7", 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "modified": "2024-09-15T03:22:33.292665Z", 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "created": "2024-09-15T03:22:33.292665Z", 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "min_mon_release": 17, 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "election_strategy": 1, 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "disallowed_leaders: ": "", 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode": false, 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "tiebreaker_mon": "", 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "removed_ranks: ": "", 2024-09-15T03:23:17.245 INFO:tasks.workunit.client.0.smithi137.stdout: "features": { 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "persistent": [ 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "kraken", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "luminous", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "mimic", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap-prune", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "nautilus", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "octopus", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "pacific", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "elector-pinging", 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "quincy" 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.246 INFO:tasks.workunit.client.0.smithi137.stdout: "optional": [] 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "mons": [ 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "rank": 0, 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "a", 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addrs": { 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:7109", 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 0 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.247 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:7109/0", 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: "priority": 0, 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 0, 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_location": "{}" 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: "quorum": [ 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: 0 2024-09-15T03:23:17.248 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap": { 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 12, 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "fsid": "145a3ccf-df94-4947-9f60-2f1acb4777f7", 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "created": "2024-09-15T03:22:34.184704+0000", 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "modified": "2024-09-15T03:23:07.339117+0000", 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "last_up_change": "2024-09-15T03:23:03.920071+0000", 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "last_in_change": "2024-09-15T03:22:34.855335+0000", 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "flags_num": 5799936, 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "flags_set": [ 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "pglog_hardlimit", 2024-09-15T03:23:17.249 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snapdirs", 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_deletes", 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "sortbitwise" 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_version": 3, 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_snapshot": "", 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_max": 1, 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "max_osd": 1, 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:23:17.250 INFO:tasks.workunit.client.0.smithi137.stdout: "min_compat_client": "jewel", 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "require_osd_release": "quincy", 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "pools": [ 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_name": "rbd", 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "create_time": "2024-09-15T03:23:05.596401+0000", 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "flags": 8193, 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "type": 1, 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "size": 2, 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "min_size": 1, 2024-09-15T03:23:17.251 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_rule": 0, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "object_hash": 2, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_num": 4, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_placement_num": 4, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_placement_num_target": 4, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_num_target": 4, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_num_pending": 4, 2024-09-15T03:23:17.252 INFO:tasks.workunit.client.0.smithi137.stdout: "last_pg_merge_meta": { 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "source_pgid": "0.0", 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "ready_epoch": 0, 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 0, 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 0, 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "source_version": "0'0", 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "target_version": "0'0" 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "12", 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "last_force_op_resend": "0", 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "auid": 0, 2024-09-15T03:23:17.253 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_seq": 3, 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_epoch": 12, 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_snaps": [], 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "removed_snaps": "[]", 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "quota_max_bytes": 0, 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "quota_max_objects": 0, 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "tiers": [], 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "tier_of": -1, 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "read_tier": -1, 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "write_tier": -1, 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_mode": "none", 2024-09-15T03:23:17.254 INFO:tasks.workunit.client.0.smithi137.stdout: "target_max_bytes": 0, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "target_max_objects": 0, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "erasure_code_profile": "", 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_params": { 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "none" 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_period": 0, 2024-09-15T03:23:17.255 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_count": 0, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "use_gmt_hitset": true, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "grade_table": [], 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "stripe_width": 0, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "expected_num_objects": 0, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "fast_read": false, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "options": {}, 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "application_metadata": { 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: "rbd": {} 2024-09-15T03:23:17.256 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "osds": [ 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "uuid": "6527c780-3561-4de7-a52a-5f7f023ba725", 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "up": 1, 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "in": 1, 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 1, 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "primary_affinity": 1, 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_begin": 0, 2024-09-15T03:23:17.257 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_end": 0, 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "up_from": 5, 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "up_thru": 7, 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "down_at": 0, 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "lost_at": 0, 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addrs": { 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6800", 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.258 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6801", 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_addrs": { 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.259 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.260 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "addrvec": [ 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v2", 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:23:17.261 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "v1", 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "nonce": 164886 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "public_addr": "127.0.0.1:6801/164886", 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "cluster_addr": "127.0.0.1:6803/164886", 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_back_addr": "127.0.0.1:6807/164886", 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "heartbeat_front_addr": "127.0.0.1:6805/164886", 2024-09-15T03:23:17.262 INFO:tasks.workunit.client.0.smithi137.stdout: "state": [ 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "exists", 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "up" 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_xinfo": [ 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "down_stamp": "0.000000", 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "laggy_probability": 0, 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "laggy_interval": 0, 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "features": 4540138320759226367, 2024-09-15T03:23:17.263 INFO:tasks.workunit.client.0.smithi137.stdout: "old_weight": 0, 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-15T03:23:03.542483+0000", 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: "dead_epoch": 0 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_upmap": [], 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_upmap_items": [], 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_temp": [], 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: "primary_temp": [], 2024-09-15T03:23:17.264 INFO:tasks.workunit.client.0.smithi137.stdout: "blocklist": {}, 2024-09-15T03:23:17.265 INFO:tasks.workunit.client.0.smithi137.stdout: "range_blocklist": {}, 2024-09-15T03:23:17.265 INFO:tasks.workunit.client.0.smithi137.stdout: "erasure_code_profiles": { 2024-09-15T03:23:17.265 INFO:tasks.workunit.client.0.smithi137.stdout: "default": { 2024-09-15T03:23:17.265 INFO:tasks.workunit.client.0.smithi137.stdout: "k": "2", 2024-09-15T03:23:17.265 INFO:tasks.workunit.client.0.smithi137.stdout: "m": "2", 2024-09-15T03:23:17.265 INFO:tasks.workunit.client.0.smithi137.stdout: "plugin": "jerasure", 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: "technique": "reed_sol_van" 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: "removed_snaps_queue": [ 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: "snaps": [ 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: "begin": 2, 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: "length": 1 2024-09-15T03:23:17.266 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: "new_removed_snaps": [ 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: "snaps": [ 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: "begin": 2, 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: "length": 1 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.267 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "new_purged_snaps": [], 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "crush_node_flags": {}, 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "device_class_flags": {}, 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode": { 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.268 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_metadata": [ 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "arch": "x86_64", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "back_addr": "[v2:127.0.0.1:6802/164886,v1:127.0.0.1:6803/164886]", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "back_iface": "", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs": "1", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:23:17.269 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:23:17.270 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_release": "quincy", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "created_at": "2024-09-15T03:22:39.670675Z", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "default_device_class": "hdd", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XLAR", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "devices": "sda", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "distro": "centos", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:23:17.271 INFO:tasks.workunit.client.0.smithi137.stdout: "distro_version": "9", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "front_addr": "[v2:127.0.0.1:6800/164886,v1:127.0.0.1:6801/164886]", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "front_iface": "", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/164886,v1:127.0.0.1:6807/164886]", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/164886,v1:127.0.0.1:6805/164886]", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "hostname": "smithi137", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "journal_rotational": "1", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "mem_swap_kb": "0", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "mem_total_kb": "32487208", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:23:17.272 INFO:tasks.workunit.client.0.smithi137.stdout: "os": "Linux", 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "osdspec_affinity": "", 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "rotational": "1" 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap_clean_epochs": { 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": { 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "per_pool": [] 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_epochs": [ 2024-09-15T03:23:17.273 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 5 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap_first_committed": 1, 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: "osdmap_last_committed": 12, 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: "crushmap": { 2024-09-15T03:23:17.274 INFO:tasks.workunit.client.0.smithi137.stdout: "devices": [ 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "osd.0", 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: "class": "hdd" 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: "types": [ 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 0, 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "osd" 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.275 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 1, 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "host" 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 2, 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "chassis" 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 3, 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "rack" 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.276 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 4, 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "row" 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 5, 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "pdu" 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 6, 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "pod" 2024-09-15T03:23:17.277 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 7, 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "room" 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 8, 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "datacenter" 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 9, 2024-09-15T03:23:17.278 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "zone" 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 10, 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "region" 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 11, 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "root" 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: "buckets": [ 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.279 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -1, 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "default", 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 11, 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "root", 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -3, 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:23:17.280 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -2, 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "default~hdd", 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 11, 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "root", 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.281 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -4, 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -3, 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "smithi137", 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 1, 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "host", 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.282 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.283 INFO:tasks.workunit.client.0.smithi137.stdout: "id": -4, 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "smithi137~hdd", 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "type_id": 1, 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "type_name": "host", 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "alg": "straw2", 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": "rjenkins1", 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "items": [ 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "id": 0, 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "weight": 6402, 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: "pos": 0 2024-09-15T03:23:17.284 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "rules": [ 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "rule_id": 0, 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "rule_name": "replicated_rule", 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "type": 1, 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "steps": [ 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "op": "take", 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "item": -1, 2024-09-15T03:23:17.285 INFO:tasks.workunit.client.0.smithi137.stdout: "item_name": "default" 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: "op": "choose_firstn", 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: "num": 0, 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: "type": "osd" 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: "op": "emit" 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.286 INFO:tasks.workunit.client.0.smithi137.stdout: "tunables": { 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_local_tries": 0, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_total_tries": 50, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "chooseleaf_stable": 1, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "straw_calc_version": 1, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "profile": "jewel", 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "optimal_tunables": 1, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "legacy_tunables": 0, 2024-09-15T03:23:17.287 INFO:tasks.workunit.client.0.smithi137.stdout: "minimum_required_version": "jewel", 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables": 1, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables2": 1, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v2_rules": 0, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables3": 1, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v3_rules": 0, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v4_buckets": 1, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "require_feature_tunables5": 1, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "has_v5_rules": 0 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "choose_args": {} 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "fsmap": { 2024-09-15T03:23:17.288 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 1, 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "default_fscid": -1, 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "compat": { 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "compat": {}, 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "ro_compat": {}, 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "incompat": { 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_1": "base v0.20", 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_8": "no anchor table", 2024-09-15T03:23:17.289 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_9": "file layout v2", 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "feature_flags": { 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "enable_multiple": true, 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "ever_enabled_multiple": true 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "standbys": [], 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "filesystems": [] 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:23:17.290 INFO:tasks.workunit.client.0.smithi137.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "auth": { 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "first_committed": 1, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "last_committed": 3, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "num_secrets": 1 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg": 4, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_active": 4, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_unknown": 0, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "num_osd": 1, 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_sum": { 2024-09-15T03:23:17.291 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 19, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 2, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 4, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 2, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 2, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-09-15T03:23:17.292 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 2, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 2, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-09-15T03:23:17.293 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-09-15T03:23:17.294 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "store_stats": { 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:23:17.295 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 0, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 0 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 3, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 3, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "up": 4, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": 4, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "num_store_stats": 0 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_sum": { 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "up_from": 0, 2024-09-15T03:23:17.296 INFO:tasks.workunit.client.0.smithi137.stdout: "seq": 0, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pgs": 4, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "num_osds": 1, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_osds": 1, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "kb": 104857600, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used": 297280, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_data": 108, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_omap": 1, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_meta": 297150, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_avail": 104560320, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "statfs": { 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 107374182400, 2024-09-15T03:23:17.297 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 107069767680, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 110592, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 22762, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 1704, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 304281944 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_peers": [], 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "num_snap_trimming": 0, 2024-09-15T03:23:17.298 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shards_repaired": 0, 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "op_queue_age_hist": { 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "histogram": [], 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "upper_bound": 1 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "perf_stat": { 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ms": 0, 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ms": 0, 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ns": 0, 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ns": 0 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "alerts": [], 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "network_ping_times": [] 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.299 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_sum_by_class": { 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "hdd": { 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "up_from": 0, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "seq": 0, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pgs": 4, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "num_osds": 1, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_osds": 1, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "kb": 104857600, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used": 297280, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_data": 108, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_omap": 1, 2024-09-15T03:23:17.300 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_used_meta": 297150, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "kb_avail": 104560320, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "statfs": { 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 107374182400, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 107069767680, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 110592, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 22762, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 1704, 2024-09-15T03:23:17.301 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 304281944 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "hb_peers": [], 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "num_snap_trimming": 0, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shards_repaired": 0, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "op_queue_age_hist": { 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "histogram": [], 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "upper_bound": 1 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "perf_stat": { 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ms": 0, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ms": 0, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "commit_latency_ns": 0, 2024-09-15T03:23:17.302 INFO:tasks.workunit.client.0.smithi137.stdout: "apply_latency_ns": 0 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "alerts": [], 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "network_ping_times": [] 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "pool_stats": [ 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "poolid": 1, 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg": 4, 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 19, 2024-09-15T03:23:17.303 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 2, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 4, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 2, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 2, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-09-15T03:23:17.304 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 2, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 2, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-09-15T03:23:17.305 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-09-15T03:23:17.306 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "store_stats": { 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "total": 0, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "available": 0, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "internally_reserved": 0, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "allocated": 4096, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "data_stored": 19, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed": 0, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_allocated": 0, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "data_compressed_original": 0, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_allocated": 0, 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: "internal_metadata": 0 2024-09-15T03:23:17.307 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 3, 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 3, 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "up": 4, 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": 4, 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "num_store_stats": 1 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "osd_stats": [ 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: "seq": 21474836482 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.308 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_by_state": [ 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "num": 2 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized", 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "num": 2 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "num_pg_by_osd": [ 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": 0, 2024-09-15T03:23:17.309 INFO:tasks.workunit.client.0.smithi137.stdout: "num_primary_pg": 4, 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "num_acting_pg": 4, 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [ 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 1, 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": {} 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "servicemap": { 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 1, 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "modified": "2024-09-15T03:22:34.185011+0000", 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: "services": {} 2024-09-15T03:23:17.310 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-09-15T03:23:17.311 INFO:tasks.workunit.client.0.smithi137.stdout: "paxos": { 2024-09-15T03:23:17.311 INFO:tasks.workunit.client.0.smithi137.stdout: "first_committed": 1, 2024-09-15T03:23:17.311 INFO:tasks.workunit.client.0.smithi137.stdout: "last_committed": 104, 2024-09-15T03:23:17.311 INFO:tasks.workunit.client.0.smithi137.stdout: "last_pn": 0, 2024-09-15T03:23:17.311 INFO:tasks.workunit.client.0.smithi137.stdout: "accepted_pn": 0 2024-09-15T03:23:17.311 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-09-15T03:23:17.311 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:report 2654924168 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: test_wait_for_peered: run_osd td/ceph-helpers 1 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:23:17.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:17.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:23:17.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-15T03:23:17.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:23:17.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e471d519-0f98-4996-964c-2a735a29edf5 2024-09-15T03:23:17.318 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 e471d519-0f98-4996-964c-2a735a29edf5 2024-09-15T03:23:17.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e471d519-0f98-4996-964c-2a735a29edf5' 2024-09-15T03:23:17.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:23:17.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlU+ZmMT8FEBAAEJ8wiILPMBXrZ0KcQraWyA== 2024-09-15T03:23:17.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlU+ZmMT8FEBAAEJ8wiILPMBXrZ0KcQraWyA=="}' 2024-09-15T03:23:17.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e471d519-0f98-4996-964c-2a735a29edf5 -i td/ceph-helpers/1/new.json 2024-09-15T03:23:17.740 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:23:17.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-15T03:23:17.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlU+ZmMT8FEBAAEJ8wiILPMBXrZ0KcQraWyA== --osd-uuid e471d519-0f98-4996-964c-2a735a29edf5 2024-09-15T03:23:17.776 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:17.775+0000 7f3f5611c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:17.777 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:17.777+0000 7f3f5611c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:17.778 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:17.778+0000 7f3f5611c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:17.779 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:17.778+0000 7f3f5611c440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:23:17.779 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:17.778+0000 7f3f5611c440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:23:17.779 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:17.778+0000 7f3f5611c440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-15T03:23:17.822 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:17.821+0000 7f3f5611c440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-15T03:23:23.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-15T03:23:23.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:23:23.452 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:23:23.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:23:23.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:23:23.934 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:23:23.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:23:23.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:23:23.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:23:23.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:23:23.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:23:23.957 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:23.955+0000 7f2156d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:23.958 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:23.957+0000 7f2156d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:23.959 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:23.958+0000 7f2156d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:24.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:23:24.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:23:24.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:23:24.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:23:24.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:23:24.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:24.285 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:23:24.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:23:24.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:24.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:24.475 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:24.473+0000 7f2156d1f440 -1 Falling back to public interface 2024-09-15T03:23:24.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:25.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:25.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:25.643 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:23:25.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:23:25.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:25.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:25.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:27.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:27.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:27.000 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:23:27.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:23:27.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:27.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:27.007 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:27.006+0000 7f2156d1f440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:23:27.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:28.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:28.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:28.361 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:23:28.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:23:28.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:28.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:28.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:29.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:29.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:29.781 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:23:29.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:23:29.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:29.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:30.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:31.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:31.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:31.197 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:23:31.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:23:31.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:31.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:31.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:32.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:32.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:32.611 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:23:32.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:23:32.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:32.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:33.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:34.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:34.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:34.077 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:23:34.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:23:34.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:34.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:34.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:35.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:35.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:35.434 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:23:35.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:23:35.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:35.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:35.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:36.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:36.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:36.977 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:23:36.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:23:36.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:36.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:37.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:38.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:38.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:38.373 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:23:38.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:23:38.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:38.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:38.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:39.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:39.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:39.782 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:23:39.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:23:39.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:39.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:40.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:41.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:41.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:41.859 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:23:41.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:23:41.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:41.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:42.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:43.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:43.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:43.684 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:23:43.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:23:43.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:43.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:44.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:45.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:45.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:45.426 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:23:45.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:23:45.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:45.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:46.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:47.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:47.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:47.209 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:23:47.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:23:47.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:47.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:47.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:48.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:48.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:48.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:23:48.969 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:23:48.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:48.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:49.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:23:50.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:23:50.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:23:50.891 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:23:50.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:23:50.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:23:50.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:23:51.429 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/167446,v1:127.0.0.1:6811/167446] [v2:127.0.0.1:6812/167446,v1:127.0.0.1:6813/167446] exists,up e471d519-0f98-4996-964c-2a735a29edf5 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: test_wait_for_peered: wait_for_peered 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-15T03:23:51.431 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:23:51.432 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:23:51.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:23:51.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:23:51.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:23:51.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:23:51.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-15T03:23:51.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-15T03:23:51.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-15T03:23:51.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:23:51.517 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:23:51.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:23:51.887 INFO:tasks.workunit.client.0.smithi137.stderr:1' 2024-09-15T03:23:51.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:23:51.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:23:51.887 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:23:51.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-15T03:23:51.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-15T03:23:51.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-15T03:23:51.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:23:51.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:23:52.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476744 2024-09-15T03:23:52.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476744 2024-09-15T03:23:52.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491 1-68719476744' 2024-09-15T03:23:52.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:23:52.108 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-15T03:23:52.108 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:23:52.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:23:52.109 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-15T03:23:52.109 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:23:52.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-15T03:23:52.110 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836491 2024-09-15T03:23:52.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-15T03:23:52.110 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:23:52.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-15T03:23:52.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:23:53.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:23:53.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:23:53.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-15T03:23:53.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:23:53.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-68719476744 2024-09-15T03:23:53.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:23:53.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:23:53.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-68719476744 2024-09-15T03:23:53.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:23:53.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476744 2024-09-15T03:23:53.821 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 68719476744 2024-09-15T03:23:53.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 68719476744' 2024-09-15T03:23:53.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:23:54.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476744 -lt 68719476744 2024-09-15T03:23:54.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-15T03:23:54.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:54.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:54.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 4 == 0 2024-09-15T03:23:54.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-15T03:23:54.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-15T03:23:54.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-15T03:23:54.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-15T03:23:54.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-15T03:23:54.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-15T03:23:54.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-15T03:23:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=4 2024-09-15T03:23:55.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-15T03:23:55.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:23:55.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:23:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 4 = 4 2024-09-15T03:23:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-15T03:23:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-15T03:23:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: test_wait_for_peered: teardown td/ceph-helpers 2024-09-15T03:23:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:23:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:23:55.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:23:55.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:23:55.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:23:55.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:23:55.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:23:55.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:23:55.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:23:55.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:23:55.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:23:55.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:23:55.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:23:55.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:23:55.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:23:55.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:23:55.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:23:55.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:23:55.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:23:55.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:23:55.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:23:55.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:23:55.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:23:55.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:55.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:55.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:23:55.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:23:55.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_scrub td/ceph-helpers 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: test_wait_for_scrub: local dir=td/ceph-helpers 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_wait_for_scrub: setup td/ceph-helpers 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:23:55.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:23:55.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:23:55.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:23:55.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:23:55.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:23:55.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:23:55.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:23:55.626 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:23:55.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:23:55.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:23:55.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:23:55.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:23:55.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:23:55.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:23:55.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:23:55.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:23:55.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:23:55.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:23:55.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:23:55.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:55.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:55.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:23:55.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:23:55.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:23:55.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:23:55.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:23:55.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:55.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:55.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:23:55.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:23:55.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:23:55.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:23:55.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:23:55.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1997: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:23:55.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:23:55.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:23:55.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:23:55.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:23:55.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:23:55.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:23:55.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:23:55.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:23:55.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:23:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:23:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:55.989 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:23:55.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:23:56.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:23:56.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:23:56.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:23:56.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:23:56.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:23:56.244 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:23:56.244 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:23:56.244 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:23:56.244 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:23:56.245 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:23:56.245 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:56.245 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:56.245 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:23:56.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:23:56.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:23:56.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:23:56.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:23:56.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:23:56.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:23:56.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:23:56.318 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:23:56.318 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:23:56.318 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:23:56.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:23:56.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:56.319 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:56.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:23:56.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:23:56.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:23:56.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1998: test_wait_for_scrub: run_mgr td/ceph-helpers x 2024-09-15T03:23:56.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:23:56.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:23:56.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:23:56.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:23:56.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:23:56.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:23:57.225 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:23:57.225 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:23:57.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:23:57.226 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:23:57.226 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:57.226 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:57.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:23:57.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:23:57.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1999: test_wait_for_scrub: run_osd td/ceph-helpers 0 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:23:57.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:23:57.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:23:57.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:23:57.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:23:57.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:23:57.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:23:57.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=895e9d28-8703-4ded-9f4e-a62ed6b2bba9 2024-09-15T03:23:57.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 895e9d28-8703-4ded-9f4e-a62ed6b2bba9' 2024-09-15T03:23:57.257 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 895e9d28-8703-4ded-9f4e-a62ed6b2bba9 2024-09-15T03:23:57.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:23:57.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNU+Zm4l36DxAAvAG0Wd4YIq2qfbX+UWTIJg== 2024-09-15T03:23:57.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNU+Zm4l36DxAAvAG0Wd4YIq2qfbX+UWTIJg=="}' 2024-09-15T03:23:57.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 895e9d28-8703-4ded-9f4e-a62ed6b2bba9 -i td/ceph-helpers/0/new.json 2024-09-15T03:23:57.578 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:23:57.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:23:57.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNU+Zm4l36DxAAvAG0Wd4YIq2qfbX+UWTIJg== --osd-uuid 895e9d28-8703-4ded-9f4e-a62ed6b2bba9 2024-09-15T03:23:57.613 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:57.613+0000 7f7977129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:57.625 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:57.624+0000 7f7977129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:57.626 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:57.625+0000 7f7977129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:23:57.626 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:57.626+0000 7f7977129440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:23:57.626 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:57.626+0000 7f7977129440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:23:57.627 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:57.626+0000 7f7977129440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:23:57.665 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:23:57.664+0000 7f7977129440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:24:03.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:24:03.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:24:03.004 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:24:03.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:24:03.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:24:03.491 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:24:03.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:24:03.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:24:03.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:24:03.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:24:03.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:24:03.514 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:03.513+0000 7fb0fad21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:03.516 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:03.515+0000 7fb0fad21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:03.517 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:03.516+0000 7fb0fad21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:03.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:24:03.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:24:03.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:24:03.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:24:03.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:24:03.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:03.844 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:24:03.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:24:03.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:03.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:04.038+0000 7fb0fad21440 -1 Falling back to public interface 2024-09-15T03:24:04.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:05.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:05.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:05.213 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:24:05.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:24:05.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:05.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:05.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:06.508 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:06.508+0000 7fb0fad21440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:24:06.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:06.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:06.572 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:24:06.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:24:06.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:06.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:06.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:07.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:07.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:07.975 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:24:07.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:24:07.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:07.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:08.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:09.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:09.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:09.450 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:24:09.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:24:09.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:09.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:09.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:10.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:10.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:10.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:24:10.851 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:24:10.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:10.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:11.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:12.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:12.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:12.294 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:24:12.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:24:12.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:12.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:12.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:13.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:13.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:13.666 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:24:13.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:24:13.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:13.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:14.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:15.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:15.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:15.150 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:24:15.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:24:15.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:15.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:15.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:16.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:16.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:16.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:24:16.529 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:24:16.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:16.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:16.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:17.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:17.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:17.890 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:24:17.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:24:17.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:17.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:18.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:19.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:19.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:19.683 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:24:19.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:24:19.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:19.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:20.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:21.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:21.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:21.408 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:24:21.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:24:21.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:21.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:22.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:23.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:23.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:23.161 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:24:23.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:24:23.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:23.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:23.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:24.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:24.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:24.897 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:24:24.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:24:24.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:24.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:25.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:26.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:26.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:26.320 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:24:26.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:24:26.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:26.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:27.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:28.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:28.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:28.131 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:24:28.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:24:28.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:28.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:28.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:29.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:29.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:29.635 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:24:29.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:24:29.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:29.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:29.994 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/169751,v1:127.0.0.1:6803/169751] [v2:127.0.0.1:6804/169751,v1:127.0.0.1:6805/169751] exists,up 895e9d28-8703-4ded-9f4e-a62ed6b2bba9 2024-09-15T03:24:29.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:24:29.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:24:29.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:24:29.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2000: test_wait_for_scrub: create_rbd_pool 2024-09-15T03:24:29.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:24:30.334 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:24:30.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:24:30.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:24:30.872 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:24:30.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:24:31.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:24:33.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: test_wait_for_scrub: wait_for_clean 2024-09-15T03:24:33.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:24:33.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:24:33.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:24:33.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:24:33.014 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:24:33.015 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:24:33.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:24:33.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:24:33.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:24:33.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:24:33.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:24:33.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:24:33.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:24:33.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:24:33.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:24:33.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:24:33.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:24:33.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:24:33.458 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:24:33.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-15T03:24:33.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-15T03:24:33.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-15T03:24:33.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:24:33.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-15T03:24:33.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:24:33.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:24:33.581 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-15T03:24:33.581 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:24:33.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-15T03:24:33.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-15T03:24:33.582 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836485 2024-09-15T03:24:33.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:24:33.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-15T03:24:33.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:24:34.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:24:34.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:24:35.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-15T03:24:35.288 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:24:35.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:24:35.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:24:35.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:24:35.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:24:35.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:24:35.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:24:35.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:24:35.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:24:35.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:24:35.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:24:36.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:24:36.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:24:36.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:24:36.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:24:36.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:24:36.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:24:36.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:24:36.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: test_wait_for_scrub: local pgid=1.0 2024-09-15T03:24:36.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2003: test_wait_for_scrub: ceph pg repair 1.0 2024-09-15T03:24:37.082 INFO:tasks.workunit.client.0.smithi137.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-15T03:24:37.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2004: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-15T03:24:37.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:37.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:37.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:37.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:37.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2004: test_wait_for_scrub: local last_scrub=2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:37.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:37.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-15T03:24:37.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:37.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T03:24:37.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-15T03:24:37.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:24:37.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:37.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:37.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:37.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:37.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:37.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:30.623522+0000 '>' 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:37.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:24:38.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:24:38.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:24:38.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:38.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:38.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:38.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:38.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:39.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:30.623522+0000 '>' 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:39.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:24:40.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:24:40.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:24:40.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:40.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:40.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:40.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:40.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:40.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:30.623522+0000 '>' 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:40.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:24:41.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:24:41.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:24:41.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:41.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:41.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:41.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:41.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:41.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:30.623522+0000 '>' 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:41.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:24:42.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:24:42.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:24:42.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:42.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:42.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:42.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:42.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:43.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:30.623522+0000 '>' 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:43.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:24:44.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:24:44.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:24:44.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:44.181 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:44.181 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:44.181 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:44.181 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:44.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:30.623522+0000 '>' 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:44.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:24:45.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:24:45.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-15T03:24:45.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:45.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:45.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:45.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:45.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:45.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:39.383938+0000 '>' 2024-09-15T03:24:30.623522+0000 2024-09-15T03:24:45.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-15T03:24:45.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-15T03:24:45.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:24:45.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:24:45.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:24:45.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:24:45.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:24:45.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:24:45.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-15T03:24:45.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:45.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:45.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:45.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:46.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_wait_for_scrub: last_scrub=2024-09-15T03:24:39.383938+0000 2024-09-15T03:24:46.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_wait_for_scrub: TIMEOUT=1 2024-09-15T03:24:46.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-15T03:24:39.383938+0000 2024-09-15T03:24:46.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-15T03:24:46.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-15T03:24:39.383938+0000 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T03:24:46.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T03:24:46.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-15T03:24:39.383938+0000 '>' 2024-09-15T03:24:39.383938+0000 2024-09-15T03:24:46.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: wait_for_scrub: return 1 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: test_wait_for_scrub: teardown td/ceph-helpers 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:24:47.673 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:24:47.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:24:47.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:24:47.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:24:47.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:24:47.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:24:47.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:24:47.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:24:47.782 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:24:47.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:24:47.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:24:47.784 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:24:47.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:24:47.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:24:47.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:24:47.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:24:47.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:24:47.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:24:47.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:24:47.795 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:24:47.795 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:47.795 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:47.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:24:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:24:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:24:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-15T03:24:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2024-09-15T03:24:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-15T03:24:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:24:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:24:47.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:24:47.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:24:47.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:24:47.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:24:47.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:24:47.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:24:47.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:24:47.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:24:47.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:24:47.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:24:47.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:24:47.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:47.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:47.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:24:47.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:24:47.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:24:47.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-15T03:24:47.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:24:47.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:47.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:47.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77294 2024-09-15T03:24:47.811 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:24:47.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:24:47.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:24:47.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-15T03:24:47.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:24:47.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-15T03:24:47.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:24:47.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:24:47.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:24:47.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-15T03:24:47.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:24:48.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:24:48.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:24:48.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:24:48.172 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:24:48.172 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:48.172 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:48.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:24:48.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T03:24:48.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:24:48.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:24:48.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:24:48.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:24:48.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:24:48.426 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get fsid 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:24:48.511 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:24:48.512 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:48.512 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:48.512 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77294/ceph-mon.a.asok 2024-09-15T03:24:48.512 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:24:48.512 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77294/ceph-mon.a.asok config get mon_host 2024-09-15T03:24:48.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-15T03:24:48.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-15T03:24:48.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:24:48.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:24:48.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:24:48.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-15T03:24:48.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:24:49.434 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:24:49.434 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:24:49.434 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:24:49.434 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:24:49.435 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:49.435 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:49.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:24:49.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:24:49.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:24:49.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-15T03:24:49.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-15T03:24:49.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' 2024-09-15T03:24:49.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-15T03:24:49.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:24:49.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8f53e9e-ebcb-42d6-9502-8d5b030c2d20 2024-09-15T03:24:49.465 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 c8f53e9e-ebcb-42d6-9502-8d5b030c2d20 2024-09-15T03:24:49.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c8f53e9e-ebcb-42d6-9502-8d5b030c2d20' 2024-09-15T03:24:49.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:24:49.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBU+ZmX4xZHBAAESeRjzIShr1nLrYZkbACPQ== 2024-09-15T03:24:49.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBU+ZmX4xZHBAAESeRjzIShr1nLrYZkbACPQ=="}' 2024-09-15T03:24:49.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8f53e9e-ebcb-42d6-9502-8d5b030c2d20 -i td/ceph-helpers/0/new.json 2024-09-15T03:24:49.796 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:24:49.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-15T03:24:49.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBU+ZmX4xZHBAAESeRjzIShr1nLrYZkbACPQ== --osd-uuid c8f53e9e-ebcb-42d6-9502-8d5b030c2d20 2024-09-15T03:24:49.829 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:49.827+0000 7f4244e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:49.840 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:49.839+0000 7f4244e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:49.841 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:49.840+0000 7f4244e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:49.842 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:49.840+0000 7f4244e09440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:24:49.842 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:49.840+0000 7f4244e09440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:24:49.842 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:49.840+0000 7f4244e09440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-15T03:24:49.881 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:49.879+0000 7f4244e09440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-15T03:24:55.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-15T03:24:55.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:24:55.230 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:24:55.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:24:55.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:24:55.718 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:24:55.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:24:55.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=145a3ccf-df94-4947-9f60-2f1acb4777f7 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77294/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:24:55.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:24:55.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:24:55.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:24:55.741 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:55.740+0000 7fd2c3065440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:55.743 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:55.742+0000 7fd2c3065440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:55.744 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:55.743+0000 7fd2c3065440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:24:56.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:24:56.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:24:56.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:24:56.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:24:56.069 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:24:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:24:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:24:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:56.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:56.271 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:56.271+0000 7fd2c3065440 -1 Falling back to public interface 2024-09-15T03:24:56.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:57.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:57.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:57.436 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:24:57.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:24:57.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:57.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:57.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:24:58.750 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:24:58.749+0000 7fd2c3065440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:24:58.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:24:58.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:24:58.787 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:24:58.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:24:58.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:24:58.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:24:59.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:00.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:00.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:00.231 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:25:00.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:25:00.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:00.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:00.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:01.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:01.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:01.705 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:25:01.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:25:01.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:01.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:02.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:03.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:03.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:03.111 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:25:03.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:25:03.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:03.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:03.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:04.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:04.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:04.608 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:25:04.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:25:04.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:04.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:05.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:06.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:06.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:06.031 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:25:06.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:25:06.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:06.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:06.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:07.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:07.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:07.422 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:25:07.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:25:07.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:07.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:07.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:08.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:08.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:08.819 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:25:08.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:25:08.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:08.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:09.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:10.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:10.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:10.224 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:25:10.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:25:10.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:10.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:10.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:11.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:11.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:11.842 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:25:11.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:25:11.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:11.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:12.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:13.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:13.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:13.623 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:25:13.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:25:13.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:13.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:14.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:15.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:15.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:15.441 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:25:15.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:25:15.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:15.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:16.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:17.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:17.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:17.497 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:25:17.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:25:17.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:17.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:18.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:19.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:19.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:19.150 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:25:19.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:25:19.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:19.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:19.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:20.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:20.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:20.639 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:25:20.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:25:20.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:20.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:21.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:22.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:22.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:22.070 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:25:22.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:25:22.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:22.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:22.421 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/172352,v1:127.0.0.1:6803/172352] [v2:127.0.0.1:6804/172352,v1:127.0.0.1:6805/172352] exists,up c8f53e9e-ebcb-42d6-9502-8d5b030c2d20 2024-09-15T03:25:22.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:25:22.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:25:22.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:25:22.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2024-09-15T03:25:22.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T03:25:22.760 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-09-15T03:25:22.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T03:25:22.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:25:23.493 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-09-15T03:25:23.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:25:24.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T03:25:25.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2024-09-15T03:25:25.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:25:25.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:25:25.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:25:25.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:25:25.598 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:25:25.598 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:25:25.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:25:25.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:25:25.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:25:25.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:25:25.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:25:25.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:25:25.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:25:25.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:25:25.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:25:26.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T03:25:26.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:25:26.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:25:26.043 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:25:26.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836490 2024-09-15T03:25:26.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836490 2024-09-15T03:25:26.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490' 2024-09-15T03:25:26.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:25:26.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836490 2024-09-15T03:25:26.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:25:26.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:25:26.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836490 2024-09-15T03:25:26.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:25:26.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836490 2024-09-15T03:25:26.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-15T03:25:26.153 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836490 2024-09-15T03:25:26.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:25:26.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-15T03:25:26.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:25:27.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:25:27.502 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:25:27.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-15T03:25:27.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:25:27.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:25:27.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:25:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T03:25:28.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:25:28.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:25:28.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:25:28.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:25:28.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:25:28.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:25:28.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:25:28.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T03:25:28.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:25:28.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:25:28.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2024-09-15T03:25:29.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2024-09-15T03:25:29.171 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:25:29.171 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-15T03:25:29.171 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2024-09-15T03:25:29.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-15T03:25:29.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-15T03:25:29.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-15T03:25:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2024-09-15T03:25:29.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-15T03:25:29.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-15T03:25:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:25:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-15T03:25:29.576 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:25:29.576 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:25:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:25:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:25:29.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:25:29.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:25:29.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:25:29.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:25:29.680 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:25:29.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:25:29.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:25:29.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:25:29.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:25:29.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:25:29.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:25:29.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:25:29.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:25:29.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:25:29.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-15T03:25:29.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:25:29.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:29.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77294 2024-09-15T03:25:29.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77294 2024-09-15T03:25:29.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:25:29.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:25:29.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: : exit 0 2024-09-15T03:25:29.693 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:25:29.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:25:29.721 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2024-09-15T03:25:29.721 DEBUG:teuthology.orchestra.run.smithi137:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-09-15T03:25:29.783 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T03:25:29.789 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T03:25:29.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:25:29.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:25:29.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T03:25:29.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T03:25:29.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-snaptrim-stats 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-15T03:25:29.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2024-09-15T03:25:29.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2024-09-15T03:25:29.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none ' 2024-09-15T03:25:29.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-15T03:25:29.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-15T03:25:29.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2024-09-15T03:25:29.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T03:25:29.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_snaptrim_stats_multiple_snaps' 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:25:29.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:25:29.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:25:29.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:25:29.798 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:25:29.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:25:29.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:25:29.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:25:29.799 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:25:29.799 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:29.799 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:29.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:25:29.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:25:29.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:25:29.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-15T03:25:29.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:25:29.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:29.801 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:29.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173648 2024-09-15T03:25:29.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:25:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T03:25:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T03:25:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:25:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:25:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:25:29.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:25:29.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:25:29.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:25:29.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:25:29.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:25:29.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:25:29.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:25:29.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:25:29.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:25:29.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:25:29.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:25:29.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:25:29.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:25:29.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:25:29.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:29.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:29.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:25:29.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:25:29.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:25:29.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-15T03:25:29.812 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:25:29.812 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:29.812 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:29.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173648 2024-09-15T03:25:29.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:25:29.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:25:29.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:25:29.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-15T03:25:29.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-15T03:25:29.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-15T03:25:29.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:25:29.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:25:29.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:25:29.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-15T03:25:29.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-15T03:25:30.201 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:25:30.201 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:25:30.201 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:25:30.201 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:25:30.201 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:30.201 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:30.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:25:30.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T03:25:30.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173648/ceph-mon.a.asok 2024-09-15T03:25:30.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:25:30.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173648/ceph-mon.a.asok config get fsid 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:25:30.537 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:25:30.538 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:30.538 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:30.538 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173648/ceph-mon.a.asok 2024-09-15T03:25:30.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:25:30.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173648/ceph-mon.a.asok config get mon_host 2024-09-15T03:25:30.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2024-09-15T03:25:30.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-15T03:25:30.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:25:30.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:25:30.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:25:30.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-15T03:25:30.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:25:31.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:25:31.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:25:31.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:25:31.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:25:31.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:31.447 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:31.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:25:31.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:25:31.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:25:31.469 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2024-09-15T03:25:31.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2024-09-15T03:25:31.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T03:25:31.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:25:31.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-15T03:25:31.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:25:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:25:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-15T03:25:31.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:25:31.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3bfdf3c-bec7-4fe0-98e0-35080b070160 2024-09-15T03:25:31.478 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 f3bfdf3c-bec7-4fe0-98e0-35080b070160 2024-09-15T03:25:31.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f3bfdf3c-bec7-4fe0-98e0-35080b070160' 2024-09-15T03:25:31.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:25:31.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrU+ZmkLwIHRAAJ9lDEhve0rGgLrAqyv94jA== 2024-09-15T03:25:31.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrU+ZmkLwIHRAAJ9lDEhve0rGgLrAqyv94jA=="}' 2024-09-15T03:25:31.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3bfdf3c-bec7-4fe0-98e0-35080b070160 -i td/test-snaptrim-stats/0/new.json 2024-09-15T03:25:31.790 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:25:31.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-15T03:25:31.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCrU+ZmkLwIHRAAJ9lDEhve0rGgLrAqyv94jA== --osd-uuid f3bfdf3c-bec7-4fe0-98e0-35080b070160 2024-09-15T03:25:31.824 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:31.822+0000 7fcda6837440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:25:31.837 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:31.835+0000 7fcda6837440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:25:31.838 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:31.836+0000 7fcda6837440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:25:31.838 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:31.836+0000 7fcda6837440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-15T03:25:31.838 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:31.837+0000 7fcda6837440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-15T03:25:31.838 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:31.837+0000 7fcda6837440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-15T03:25:31.877 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:31.875+0000 7fcda6837440 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-15T03:25:37.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-15T03:25:37.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:25:37.417 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:25:37.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:25:37.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:25:37.898 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:25:37.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:25:37.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:25:37.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:25:37.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:25:37.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:25:37.920 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:37.918+0000 7f6685616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:25:37.922 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:37.920+0000 7f6685616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:25:37.923 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:37.921+0000 7f6685616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:25:38.246 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:25:38.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:38.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:38.444 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:38.442+0000 7f6685616440 -1 Falling back to public interface 2024-09-15T03:25:38.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:39.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:39.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:39.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:25:39.601 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:25:39.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:39.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:39.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:25:40.937+0000 7f6685616440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:25:40.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:40.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:40.956 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:25:40.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:25:40.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:40.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:41.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:42.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:42.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:42.323 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:25:42.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:25:42.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:42.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:42.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:43.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:43.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:43.787 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:25:43.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:25:43.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:43.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:44.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:45.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:45.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:45.185 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:25:45.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:25:45.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:45.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:45.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:46.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:46.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:46.693 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:25:46.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:25:46.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:46.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:47.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:48.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:48.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:48.114 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:25:48.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:25:48.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:48.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:48.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:49.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:49.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:49.517 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:25:49.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:25:49.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:49.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:49.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:50.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:50.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:50.943 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:25:50.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:25:50.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:50.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:51.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:52.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:52.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:52.305 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:25:52.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:25:52.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:52.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:52.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:53.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:53.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:53.787 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:25:53.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:25:53.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:53.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:54.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:55.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:55.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:55.773 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:25:55.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:25:55.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:55.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:56.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:57.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:57.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:57.411 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:25:57.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:25:57.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:57.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:57.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:25:58.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:25:58.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:25:58.986 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:25:58.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:25:58.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:25:58.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:25:59.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:00.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:00.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:00.548 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:26:00.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:26:00.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:00.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:26:01.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:02.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:02.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:02.290 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:26:02.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:26:02.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:02.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:26:03.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:04.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:04.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:04.193 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:26:04.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:26:04.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:04.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:26:04.647 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174758,v1:127.0.0.1:6803/174758] [v2:127.0.0.1:6804/174758,v1:127.0.0.1:6805/174758] exists,up f3bfdf3c-bec7-4fe0-98e0-35080b070160 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:26:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:26:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-15T03:26:04.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61834076-ef07-429f-a2b6-05309e89381b 2024-09-15T03:26:04.654 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 61834076-ef07-429f-a2b6-05309e89381b 2024-09-15T03:26:04.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 61834076-ef07-429f-a2b6-05309e89381b' 2024-09-15T03:26:04.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:26:04.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMU+ZmGqt7JxAAdNzWWM9SF55OJU/Sj4KKhQ== 2024-09-15T03:26:04.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMU+ZmGqt7JxAAdNzWWM9SF55OJU/Sj4KKhQ=="}' 2024-09-15T03:26:04.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61834076-ef07-429f-a2b6-05309e89381b -i td/test-snaptrim-stats/1/new.json 2024-09-15T03:26:05.207 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:26:05.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-15T03:26:05.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDMU+ZmGqt7JxAAdNzWWM9SF55OJU/Sj4KKhQ== --osd-uuid 61834076-ef07-429f-a2b6-05309e89381b 2024-09-15T03:26:05.242 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:05.241+0000 7f987140e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:05.257 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:05.256+0000 7f987140e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:05.258 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:05.257+0000 7f987140e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:05.258 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:05.257+0000 7f987140e440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-15T03:26:05.258 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:05.257+0000 7f987140e440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-15T03:26:05.258 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:05.257+0000 7f987140e440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-15T03:26:05.337 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:05.336+0000 7f987140e440 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-15T03:26:11.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-15T03:26:11.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:26:11.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:26:11.106 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:26:11.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:26:11.586 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:26:11.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:26:11.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:26:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:26:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:26:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:26:11.607 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:11.605+0000 7f910132c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:11.609 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:11.607+0000 7f910132c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:11.610 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:11.608+0000 7f910132c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:11.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:26:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:26:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:26:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:26:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:26:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:26:11.936 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:26:11.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:11.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:12.127 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:12.126+0000 7f910132c440 -1 Falling back to public interface 2024-09-15T03:26:12.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:13.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:13.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:13.305 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:26:13.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:26:13.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:13.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:13.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:14.594 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:14.593+0000 7f910132c440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:26:14.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:14.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:14.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:26:14.661 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:26:14.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:14.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:15.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:16.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:16.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:16.351 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:26:16.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:26:16.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:16.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:16.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:17.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:17.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:17.711 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:26:17.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:26:17.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:17.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:18.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:19.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:19.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:19.173 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:26:19.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:26:19.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:19.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:19.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:20.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:20.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:20.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:26:20.592 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:26:20.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:20.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:20.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:21.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:21.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:21.982 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:26:21.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:26:21.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:21.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:22.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:23.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:23.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:23.480 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:26:23.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:26:23.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:23.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:23.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:24.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:24.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:24.993 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:26:24.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:26:24.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:24.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:25.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:26.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:26.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:26.651 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:26:26.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:26:26.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:26.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:27.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:28.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:28.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:28.019 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:26:28.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:26:28.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:28.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:28.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:29.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:29.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:29.638 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:26:29.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:26:29.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:29.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:30.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:31.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:31.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:31.525 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:26:31.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:26:31.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:31.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:32.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:33.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:33.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:33.262 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:26:33.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:26:33.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:33.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:34.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:35.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:35.200 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:26:35.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:35.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:26:35.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:35.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:35.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:36.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:36.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:36.880 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:26:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:26:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:36.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:37.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:38.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:38.327 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:26:38.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:38.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:26:38.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:38.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:38.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:39.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:39.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:39.759 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:26:39.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:26:39.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:39.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:26:40.112 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/176517,v1:127.0.0.1:6811/176517] [v2:127.0.0.1:6812/176517,v1:127.0.0.1:6813/176517] exists,up 61834076-ef07-429f-a2b6-05309e89381b 2024-09-15T03:26:40.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:26:40.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:26:40.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-15T03:26:40.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:26:40.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d3cc1481-99ee-4ddc-92bf-7dfd07675495 2024-09-15T03:26:40.119 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 d3cc1481-99ee-4ddc-92bf-7dfd07675495 2024-09-15T03:26:40.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d3cc1481-99ee-4ddc-92bf-7dfd07675495' 2024-09-15T03:26:40.119 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:26:40.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwU+ZmqM2kBxAAZRslHtiyQCTKMOoZYa2GoQ== 2024-09-15T03:26:40.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwU+ZmqM2kBxAAZRslHtiyQCTKMOoZYa2GoQ=="}' 2024-09-15T03:26:40.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d3cc1481-99ee-4ddc-92bf-7dfd07675495 -i td/test-snaptrim-stats/2/new.json 2024-09-15T03:26:40.639 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:26:40.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-15T03:26:40.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDwU+ZmqM2kBxAAZRslHtiyQCTKMOoZYa2GoQ== --osd-uuid d3cc1481-99ee-4ddc-92bf-7dfd07675495 2024-09-15T03:26:40.675 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:40.673+0000 7f31cfd1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:40.697 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:40.696+0000 7f31cfd1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:40.698 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:40.697+0000 7f31cfd1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:40.698 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:40.697+0000 7f31cfd1d440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-15T03:26:40.699 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:40.697+0000 7f31cfd1d440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-15T03:26:40.699 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:40.697+0000 7f31cfd1d440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-15T03:26:40.789 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:40.787+0000 7f31cfd1d440 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-15T03:26:46.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-15T03:26:46.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:26:46.939 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:26:46.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:26:46.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:26:47.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:26:47.444 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:26:47.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:26:47.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:26:47.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:26:47.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:26:47.465 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:47.464+0000 7fabae382440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:47.467 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:47.466+0000 7fabae382440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:47.468 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:47.467+0000 7fabae382440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:26:47.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:26:47.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:26:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:26:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:26:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:26:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:47.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:26:47.797 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:26:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:47.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:47.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:47.981+0000 7fabae382440 -1 Falling back to public interface 2024-09-15T03:26:48.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:49.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:49.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:49.185 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:26:49.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:26:49.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:49.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:49.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:50.478 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:26:50.478+0000 7fabae382440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:26:50.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:50.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:50.539 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:26:50.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:26:50.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:50.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:50.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:51.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:51.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:51.981 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:26:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:26:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:51.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:52.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:53.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:53.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:53.445 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:26:53.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:26:53.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:53.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:53.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:54.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:54.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:54.937 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:26:54.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:26:54.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:54.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:55.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:56.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:56.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:56.461 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:26:56.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:26:56.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:56.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:56.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:57.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:57.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:57.853 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:26:57.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:26:57.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:57.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:58.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:26:59.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:26:59.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:26:59.228 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:26:59.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:26:59.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:26:59.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:26:59.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:00.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:00.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:00.604 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:27:00.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:27:00.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:00.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:00.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:01.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:01.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:01.987 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:27:01.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:27:01.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:01.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:02.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:03.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:03.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:03.374 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:27:03.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:27:03.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:03.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:04.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:05.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:05.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:05.298 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:27:05.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:27:05.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:05.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:06.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:07.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:07.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:07.198 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:27:07.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:27:07.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:07.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:08.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:09.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:09.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:09.077 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:27:09.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:27:09.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:09.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:09.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:10.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:10.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:10.588 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:27:10.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:27:10.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:10.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:11.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:12.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:12.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:12.319 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:27:12.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:27:12.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:12.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:12.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:13.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:13.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:13.776 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:27:13.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:27:13.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:13.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:14.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:15.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:15.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:15.206 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:27:15.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:27:15.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:15.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:27:15.555 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/178296,v1:127.0.0.1:6819/178296] [v2:127.0.0.1:6820/178296,v1:127.0.0.1:6821/178296] exists,up d3cc1481-99ee-4ddc-92bf-7dfd07675495 2024-09-15T03:27:15.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:27:15.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:27:15.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:27:15.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2024-09-15T03:27:16.262 INFO:tasks.workunit.client.0.smithi137.stderr:noscrub is set 2024-09-15T03:27:16.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2024-09-15T03:27:16.907 INFO:tasks.workunit.client.0.smithi137.stderr:nodeep-scrub is set 2024-09-15T03:27:16.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2024-09-15T03:27:16.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-15T03:27:17.475 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'test' already exists 2024-09-15T03:27:17.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:27:18.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2024-09-15T03:27:18.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:27:18.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:27:18.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:27:18.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:27:18.488 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:27:18.488 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:27:18.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:27:18.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:27:18.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:27:18.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:27:18.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:27:18.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:27:18.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:27:18.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:27:18.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:27:19.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:27:19.026 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T03:27:19.026 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T03:27:19.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:27:19.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:27:19.027 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:27:19.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-15T03:27:19.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-15T03:27:19.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-15T03:27:19.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:27:19.196 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:27:19.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672971 2024-09-15T03:27:19.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672971 2024-09-15T03:27:19.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672971' 2024-09-15T03:27:19.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:27:19.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:27:19.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509447 2024-09-15T03:27:19.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509447 2024-09-15T03:27:19.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672971 2-64424509447' 2024-09-15T03:27:19.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:27:19.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-15T03:27:19.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:27:19.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:27:19.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-15T03:27:19.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:27:19.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-15T03:27:19.411 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836500 2024-09-15T03:27:19.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-15T03:27:19.411 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:27:19.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-15T03:27:19.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:27:20.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:27:20.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:27:21.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-15T03:27:21.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:27:21.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672971 2024-09-15T03:27:21.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:27:21.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:27:21.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672971 2024-09-15T03:27:21.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:27:21.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672971 2024-09-15T03:27:21.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-15T03:27:21.120 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672971 2024-09-15T03:27:21.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672971 2024-09-15T03:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509447 2024-09-15T03:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:27:21.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509447 2024-09-15T03:27:21.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:27:21.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509447 2024-09-15T03:27:21.479 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509447 2024-09-15T03:27:21.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-15T03:27:21.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:27:21.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509447 2024-09-15T03:27:21.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:27:21.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:27:21.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:27:22.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T03:27:22.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:27:22.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:27:22.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:27:22.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:27:22.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:27:22.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:27:22.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:27:22.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T03:27:22.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:27:22.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:27:22.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:27:23.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T03:27:23.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:27:23.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:27:23.239 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2024-09-15T03:27:23.239 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2024-09-15T03:27:23.239 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2024-09-15T03:27:23.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2024-09-15T03:27:23.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2024-09-15T03:27:23.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-15T03:27:23.631 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-09-15T03:27:23.631 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-09-15T03:27:23.631 INFO:tasks.workunit.client.0.smithi137.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.1708e-05 s, 79.2 MB/s 2024-09-15T03:27:23.631 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2024-09-15T03:27:23.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:23.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2024-09-15T03:27:23.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:23.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2024-09-15T03:27:23.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:23.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2024-09-15T03:27:24.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:24.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2024-09-15T03:27:24.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:24.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2024-09-15T03:27:24.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:24.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2024-09-15T03:27:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:24.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2024-09-15T03:27:24.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:24.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2024-09-15T03:27:24.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:24.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2024-09-15T03:27:24.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:24.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2024-09-15T03:27:25.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2024-09-15T03:27:25.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2024-09-15T03:27:25.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2024-09-15T03:27:25.237 INFO:tasks.workunit.client.0.smithi137.stdout:created pool test snap snap1 2024-09-15T03:27:25.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2024-09-15T03:27:25.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-15T03:27:25.240 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-09-15T03:27:25.240 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-09-15T03:27:25.240 INFO:tasks.workunit.client.0.smithi137.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.989e-05 s, 82.1 MB/s 2024-09-15T03:27:25.240 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2024-09-15T03:27:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2024-09-15T03:27:25.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:25.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2024-09-15T03:27:25.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:25.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2024-09-15T03:27:25.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:25.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2024-09-15T03:27:25.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:25.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2024-09-15T03:27:26.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:26.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2024-09-15T03:27:26.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:26.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2024-09-15T03:27:26.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:26.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2024-09-15T03:27:26.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:26.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2024-09-15T03:27:26.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-15T03:27:26.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2024-09-15T03:27:26.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2024-09-15T03:27:26.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2024-09-15T03:27:26.974 INFO:tasks.workunit.client.0.smithi137.stdout:removed pool test snap snap1 2024-09-15T03:27:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2024-09-15T03:27:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:27:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:27:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:27:26.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:27:26.977 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:27:26.977 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:27:26.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:27:26.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:27:26.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:27:27.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:27:27.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:27:27.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:27:27.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:27:27.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:27:27.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:27:27.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:27:27.462 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T03:27:27.463 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T03:27:27.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:27:27.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:27:27.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:27:27.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836504 2024-09-15T03:27:27.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836504 2024-09-15T03:27:27.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504' 2024-09-15T03:27:27.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:27:27.671 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:27:27.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-15T03:27:27.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-15T03:27:27.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672975' 2024-09-15T03:27:27.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:27:27.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:27:27.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-15T03:27:27.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-15T03:27:27.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509451' 2024-09-15T03:27:27.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:27:27.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836504 2024-09-15T03:27:27.950 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:27:27.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:27:27.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836504 2024-09-15T03:27:27.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:27:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836504 2024-09-15T03:27:27.952 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836504 2024-09-15T03:27:27.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-15T03:27:27.953 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:27:28.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-15T03:27:28.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:27:29.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:27:29.336 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:27:29.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-15T03:27:29.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:27:29.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-15T03:27:29.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:27:29.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:27:29.690 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-15T03:27:29.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:27:29.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-15T03:27:29.692 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672975 2024-09-15T03:27:29.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-15T03:27:29.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:27:30.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-15T03:27:30.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:27:30.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-15T03:27:30.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:27:30.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:27:30.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-15T03:27:30.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:27:30.087 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509451 2024-09-15T03:27:30.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-15T03:27:30.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T03:27:30.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:27:30.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T03:27:30.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:27:30.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:27:30.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:27:30.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T03:27:30.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:27:30.904 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:27:30.904 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:27:30.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:27:30.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:27:30.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:27:30.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:27:31.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T03:27:31.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:27:31.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:27:31.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:27:31.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T03:27:31.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:27:31.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:27:31.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2024-09-15T03:27:41.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2024-09-15T03:27:41.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2024-09-15T03:27:41.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2024-09-15T03:27:41.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2024-09-15T03:27:41.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:41.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2024-09-15T03:27:41.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-15T03:27:41.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:41.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2024-09-15T03:27:41.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2024-09-15T03:27:41.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:41.860 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-15T03:27:41.860 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:41.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.49539139700000001 2024-09-15T03:27:41.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.49539139700000001 2024-09-15T03:27:41.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:41.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2024-09-15T03:27:41.971 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-15T03:27:41.972 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:42.079 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2024-09-15T03:27:42.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-15T03:27:42.081 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:42.081 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-15T03:27:42.081 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:42.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .49539139700000001 + 0.43669445600000001 2024-09-15T03:27:42.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.93208585300000002 2024-09-15T03:27:42.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:42.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2024-09-15T03:27:42.193 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-15T03:27:42.193 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:42.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2024-09-15T03:27:42.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-15T03:27:42.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:42.302 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-15T03:27:42.302 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:42.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .93208585300000002 + 0.00027308600000000002 2024-09-15T03:27:42.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.93235893900000002002 2024-09-15T03:27:42.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:42.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2024-09-15T03:27:42.411 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-15T03:27:42.411 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:42.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2024-09-15T03:27:42.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2024-09-15T03:27:42.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:42.521 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-15T03:27:42.522 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:42.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .93235893900000002002 + 0.73254778099999995 2024-09-15T03:27:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.66490671999999997002 2024-09-15T03:27:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2024-09-15T03:27:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-15T03:27:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:42.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2024-09-15T03:27:42.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2024-09-15T03:27:42.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:42.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-15T03:27:42.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:42.851 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.66490671999999997002 + 0.36112747699999997 2024-09-15T03:27:42.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.02603419699999994002 2024-09-15T03:27:42.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:42.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2024-09-15T03:27:42.853 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-15T03:27:42.853 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:42.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2024-09-15T03:27:42.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-15T03:27:42.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:42.964 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-15T03:27:42.964 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:43.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.02603419699999994002 + 0.43670029500000002 2024-09-15T03:27:43.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.46273449199999996002 2024-09-15T03:27:43.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:43.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2024-09-15T03:27:43.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-15T03:27:43.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:43.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2024-09-15T03:27:43.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-15T03:27:43.208 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:43.209 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-15T03:27:43.209 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:43.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.46273449199999996002 + 0.78071338300000004 2024-09-15T03:27:43.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.24344787500000000002 2024-09-15T03:27:43.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:27:43.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2024-09-15T03:27:43.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-15T03:27:43.319 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-15T03:27:43.427 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2024-09-15T03:27:43.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2024-09-15T03:27:43.429 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-15T03:27:43.430 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-15T03:27:43.430 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-15T03:27:43.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 3.24344787500000000002 + 0.95755898800000006 2024-09-15T03:27:43.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=4.20100686300000006002 2024-09-15T03:27:43.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2024-09-15T03:27:43.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '4.20100686300000006002 > 0.0' 2024-09-15T03:27:43.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2024-09-15T03:27:43.540 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:27:43.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:27:43.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:27:43.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:27:43.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:27:43.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:27:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:27:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:27:43.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:27:43.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:27:43.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:27:43.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:27:43.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:27:43.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:27:43.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:27:43.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:43.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:43.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:27:43.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:27:43.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:27:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:27:43.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:27:43.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:27:43.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:27:43.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:27:43.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:27:43.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:27:43.715 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:27:43.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:27:43.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:27:43.718 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:27:43.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:27:43.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:27:43.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:27:43.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:43.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:43.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:27:43.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:27:43.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:27:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:27:43.724 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:27:43.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:27:43.724 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:27:43.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:27:43.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:27:43.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:27:43.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:27:43.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:27:43.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:27:43.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:27:43.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:27:43.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:27:43.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:43.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:43.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:27:43.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:27:43.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:27:43.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-15T03:27:43.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:27:43.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:43.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:43.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173648 2024-09-15T03:27:43.734 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2024-09-15T03:27:43.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:27:43.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:27:43.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:27:43.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:27:43.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:27:43.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:27:43.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:27:43.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:27:43.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:27:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:27:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:27:43.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:27:43.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:27:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:27:43.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:27:43.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:27:43.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:43.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:43.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:27:43.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:27:43.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:27:43.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-15T03:27:43.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:27:43.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:43.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:43.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173648 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-15T03:27:43.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-15T03:27:44.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:27:44.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:27:44.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:27:44.101 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:27:44.101 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:44.101 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:44.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:27:44.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T03:27:44.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:27:44.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:27:44.345 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:27:44.345 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:44.345 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:44.345 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173648/ceph-mon.a.asok 2024-09-15T03:27:44.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:27:44.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173648/ceph-mon.a.asok config get fsid 2024-09-15T03:27:44.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:27:44.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:27:44.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:27:44.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:27:44.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:27:44.420 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:27:44.420 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:27:44.420 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:27:44.420 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:27:44.420 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:44.420 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:44.420 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173648/ceph-mon.a.asok 2024-09-15T03:27:44.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:27:44.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173648/ceph-mon.a.asok config get mon_host 2024-09-15T03:27:44.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2024-09-15T03:27:44.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-15T03:27:44.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:27:44.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:27:44.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:27:44.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-15T03:27:44.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:27:45.322 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:27:45.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:27:45.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:27:45.323 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:27:45.323 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:45.323 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:45.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:27:45.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:27:45.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:27:45.346 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2024-09-15T03:27:45.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2024-09-15T03:27:45.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T03:27:45.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:27:45.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-15T03:27:45.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:27:45.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-15T03:27:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:27:45.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:27:45.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27396050-49d5-4f39-9cb0-1b3667342483 2024-09-15T03:27:45.355 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 27396050-49d5-4f39-9cb0-1b3667342483 2024-09-15T03:27:45.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 27396050-49d5-4f39-9cb0-1b3667342483' 2024-09-15T03:27:45.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:27:45.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxVOZmOcmsFRAA47+OKV0RoNV6XqKKmzwbcQ== 2024-09-15T03:27:45.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxVOZmOcmsFRAA47+OKV0RoNV6XqKKmzwbcQ=="}' 2024-09-15T03:27:45.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27396050-49d5-4f39-9cb0-1b3667342483 -i td/test-snaptrim-stats/0/new.json 2024-09-15T03:27:45.667 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:27:45.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-15T03:27:45.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAxVOZmOcmsFRAA47+OKV0RoNV6XqKKmzwbcQ== --osd-uuid 27396050-49d5-4f39-9cb0-1b3667342483 2024-09-15T03:27:45.702 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:45.701+0000 7f9eb560b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:27:45.713 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:45.712+0000 7f9eb560b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:27:45.714 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:45.713+0000 7f9eb560b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:27:45.714 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:45.714+0000 7f9eb560b440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-15T03:27:45.714 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:45.714+0000 7f9eb560b440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-15T03:27:45.715 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:45.714+0000 7f9eb560b440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-15T03:27:45.753 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:45.753+0000 7f9eb560b440 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-15T03:27:51.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-15T03:27:51.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:27:51.110 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:27:51.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:27:51.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:27:51.585 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:27:51.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:27:51.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:27:51.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:27:51.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:27:51.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:27:51.607 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:51.606+0000 7f275ff57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:27:51.608 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:51.608+0000 7f275ff57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:27:51.609 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:51.609+0000 7f275ff57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:27:51.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:27:51.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:27:51.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:27:51.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:27:51.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:27:51.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:51.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:27:51.932 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:27:51.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:51.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:27:52.130 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:52.130+0000 7f275ff57440 -1 Falling back to public interface 2024-09-15T03:27:52.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:53.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:53.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:53.301 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:27:53.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:27:53.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:53.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:27:53.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:54.607 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:27:54.606+0000 7f275ff57440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:27:54.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:54.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:54.651 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:27:54.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:27:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:27:55.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:56.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:56.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:56.037 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:27:56.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:27:56.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:56.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:27:56.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:57.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:57.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:57.446 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:27:57.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:27:57.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:57.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:27:57.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:27:58.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:27:58.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:27:58.837 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:27:58.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:27:58.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:27:58.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:27:59.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:00.361 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:00.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:01.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:01.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:01.736 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:28:01.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:28:01.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:01.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:02.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:03.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:03.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:03.112 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:28:03.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:28:03.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:03.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:03.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:04.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:04.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:04.534 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:28:04.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:28:04.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:04.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:04.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:05.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:05.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:05.927 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:28:05.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:28:05.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:05.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:06.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:07.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:07.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:07.431 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:28:07.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:28:07.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:07.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:07.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:08.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:08.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:08.873 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:28:08.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:28:08.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:08.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:09.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:10.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:10.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:10.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:28:10.753 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:28:10.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:10.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:11.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:12.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:12.224 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:28:12.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:12.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:28:12.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:12.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:12.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:13.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:13.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:13.931 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:28:13.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:28:13.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:13.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:14.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:15.735 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:28:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:28:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:15.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:16.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:17.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:17.358 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:28:17.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:28:17.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:17.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:28:17.736 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/182027,v1:127.0.0.1:6803/182027] [v2:127.0.0.1:6804/182027,v1:127.0.0.1:6805/182027] exists,up 27396050-49d5-4f39-9cb0-1b3667342483 2024-09-15T03:28:17.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:28:17.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:28:17.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-15T03:28:17.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:28:17.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43907bab-5a59-4096-b61d-ca290c4b408d 2024-09-15T03:28:17.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 43907bab-5a59-4096-b61d-ca290c4b408d' 2024-09-15T03:28:17.744 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 43907bab-5a59-4096-b61d-ca290c4b408d 2024-09-15T03:28:17.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:28:17.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRVOZm0snXLBAAJTbczYij8AeXUBLEC+KiHw== 2024-09-15T03:28:17.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRVOZm0snXLBAAJTbczYij8AeXUBLEC+KiHw=="}' 2024-09-15T03:28:17.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43907bab-5a59-4096-b61d-ca290c4b408d -i td/test-snaptrim-stats/1/new.json 2024-09-15T03:28:18.266 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:28:18.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-15T03:28:18.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBRVOZm0snXLBAAJTbczYij8AeXUBLEC+KiHw== --osd-uuid 43907bab-5a59-4096-b61d-ca290c4b408d 2024-09-15T03:28:18.301 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:18.299+0000 7f67aa326440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:18.302 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:18.301+0000 7f67aa326440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:18.304 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:18.302+0000 7f67aa326440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:18.304 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:18.302+0000 7f67aa326440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-15T03:28:18.304 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:18.302+0000 7f67aa326440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-15T03:28:18.304 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:18.302+0000 7f67aa326440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-15T03:28:18.373 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:18.371+0000 7f67aa326440 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-15T03:28:24.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-15T03:28:24.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:28:24.826 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:28:24.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:28:24.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:28:25.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:28:25.306 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:28:25.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:28:25.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:28:25.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:28:25.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:28:25.328 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:25.326+0000 7f946e40b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:25.329 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:25.328+0000 7f946e40b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:25.330 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:25.329+0000 7f946e40b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:25.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:28:25.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:28:25.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:28:25.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:28:25.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:28:25.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:25.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:28:25.659 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:28:25.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:25.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:25.845 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:25.843+0000 7f946e40b440 -1 Falling back to public interface 2024-09-15T03:28:26.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:27.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:27.022 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:28:27.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:27.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:28:27.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:27.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:27.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:28.328 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:28.327+0000 7f946e40b440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:28:28.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:28.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:28.380 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:28:28.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:28:28.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:28.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:29.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:30.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:30.006 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:28:30.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:30.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:28:30.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:30.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:30.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:31.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:31.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:31.419 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:28:31.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:28:31.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:31.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:31.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:32.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:32.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:32.860 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:28:32.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:28:32.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:32.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:33.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:34.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:34.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:34.306 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:28:34.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:28:34.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:34.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:34.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:35.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:35.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:35.679 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:28:35.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:28:35.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:35.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:36.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:37.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:37.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:37.065 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:28:37.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:28:37.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:37.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:37.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:38.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:38.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:38.423 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:28:38.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:28:38.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:38.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:38.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:39.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:39.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:39.798 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:28:39.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:28:39.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:39.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:40.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:41.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:41.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:41.229 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:28:41.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:28:41.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:41.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:42.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:43.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:43.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:43.275 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:28:43.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:28:43.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:43.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:44.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:45.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:45.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:45.080 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:28:45.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:28:45.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:45.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:45.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:46.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:46.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:46.798 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:28:46.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:28:46.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:46.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:47.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:48.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:48.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:48.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:28:48.329 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:28:48.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:48.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:49.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:50.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:50.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:50.258 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:28:50.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:28:50.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:50.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:51.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:28:52.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:28:52.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:28:52.304 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:28:52.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:28:52.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:28:52.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:28:52.675 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/183781,v1:127.0.0.1:6811/183781] [v2:127.0.0.1:6812/183781,v1:127.0.0.1:6813/183781] exists,up 43907bab-5a59-4096-b61d-ca290c4b408d 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:28:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:28:52.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:28:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7c986cf-6070-4d56-8add-363ab3bfb555 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f7c986cf-6070-4d56-8add-363ab3bfb555' 2024-09-15T03:28:52.683 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 f7c986cf-6070-4d56-8add-363ab3bfb555 2024-09-15T03:28:52.684 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:28:52.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0VOZmeTonKRAAw7tXQfSpt+jLKiIuW6fjVg== 2024-09-15T03:28:52.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0VOZmeTonKRAAw7tXQfSpt+jLKiIuW6fjVg=="}' 2024-09-15T03:28:52.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7c986cf-6070-4d56-8add-363ab3bfb555 -i td/test-snaptrim-stats/2/new.json 2024-09-15T03:28:53.187 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:28:53.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-15T03:28:53.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQB0VOZmeTonKRAAw7tXQfSpt+jLKiIuW6fjVg== --osd-uuid f7c986cf-6070-4d56-8add-363ab3bfb555 2024-09-15T03:28:53.221 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:53.221+0000 7f3169e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:53.223 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:53.223+0000 7f3169e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:53.224 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:53.224+0000 7f3169e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:53.225 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:53.224+0000 7f3169e12440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-15T03:28:53.225 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:53.224+0000 7f3169e12440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-15T03:28:53.225 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:53.224+0000 7f3169e12440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-15T03:28:53.271 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:53.270+0000 7f3169e12440 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-15T03:28:59.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-15T03:28:59.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:28:59.460 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:28:59.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:28:59.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:28:59.947 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:28:59.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:28:59.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b4764645-2382-4918-b215-23d6dcd4682d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173648/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-15T03:28:59.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:28:59.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:28:59.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:28:59.969 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:59.967+0000 7f27a4009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:59.970 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:59.969+0000 7f27a4009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:28:59.971 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:28:59.970+0000 7f27a4009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:29:00.293 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:29:00.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:00.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:00.492 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:29:00.491+0000 7f27a4009440 -1 Falling back to public interface 2024-09-15T03:29:00.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:01.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:01.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:01.641 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:29:01.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:29:01.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:01.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:01.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:03.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:03.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:03.002 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:29:03.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:29:03.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:03.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:03.029 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:29:03.027+0000 7f27a4009440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:29:03.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:04.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:04.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:04.378 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:29:04.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:29:04.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:04.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:04.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:05.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:05.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:05.797 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:29:05.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:29:05.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:05.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:06.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:07.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:07.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:07.224 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:29:07.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:29:07.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:07.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:07.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:08.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:08.671 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:29:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:29:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:08.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:09.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:10.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:10.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:10.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:29:10.085 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:29:10.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:10.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:10.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:11.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:11.468 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:29:11.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:11.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:29:11.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:11.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:11.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:12.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:12.842 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:29:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:29:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:12.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:13.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:14.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:14.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:14.274 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:29:14.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:29:14.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:14.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:14.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:15.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:15.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:15.635 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:29:15.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:29:15.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:15.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:16.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:17.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:17.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:17.704 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:29:17.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:29:17.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:17.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:18.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:19.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:19.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:19.179 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:29:19.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:29:19.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:19.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:20.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:21.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:21.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:21.142 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:29:21.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:29:21.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:21.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:22.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:23.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:23.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:23.326 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:29:23.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:29:23.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:23.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:24.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:25.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:25.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:25.174 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:29:25.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:29:25.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:25.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:25.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:29:26.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:29:26.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:29:26.614 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:29:26.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:29:26.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:29:26.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:29:26.981 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/185535,v1:127.0.0.1:6819/185535] [v2:127.0.0.1:6820/185535,v1:127.0.0.1:6821/185535] exists,up f7c986cf-6070-4d56-8add-363ab3bfb555 2024-09-15T03:29:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:29:26.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:29:26.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:29:26.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2024-09-15T03:29:28.043 INFO:tasks.workunit.client.0.smithi137.stderr:noscrub is set 2024-09-15T03:29:28.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2024-09-15T03:29:28.678 INFO:tasks.workunit.client.0.smithi137.stderr:nodeep-scrub is set 2024-09-15T03:29:28.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2024-09-15T03:29:28.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-15T03:29:29.668 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'test' already exists 2024-09-15T03:29:29.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:29:30.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:29:30.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:29:30.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:29:30.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:29:30.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:29:30.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:29:30.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:29:30.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:29:31.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:29:31.201 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T03:29:31.201 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T03:29:31.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:29:31.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:31.201 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:29:31.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836499 2024-09-15T03:29:31.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836499 2024-09-15T03:29:31.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499' 2024-09-15T03:29:31.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:31.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:29:31.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-15T03:29:31.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-15T03:29:31.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672976' 2024-09-15T03:29:31.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:31.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:29:31.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509450 2024-09-15T03:29:31.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509450 2024-09-15T03:29:31.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509450' 2024-09-15T03:29:31.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:29:31.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836499 2024-09-15T03:29:31.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:29:31.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:29:31.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836499 2024-09-15T03:29:31.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:29:31.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836499 2024-09-15T03:29:31.524 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836499 2024-09-15T03:29:31.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-15T03:29:31.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:29:31.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-15T03:29:31.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:29:32.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:29:32.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:29:33.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-15T03:29:33.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:29:33.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-15T03:29:33.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:29:33.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:29:33.227 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-15T03:29:33.227 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:29:33.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-15T03:29:33.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-15T03:29:33.229 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672976 2024-09-15T03:29:33.229 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:29:33.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672976 2024-09-15T03:29:33.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:29:33.645 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509450 2024-09-15T03:29:33.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:29:33.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:29:33.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509450 2024-09-15T03:29:33.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:29:33.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509450 2024-09-15T03:29:33.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-15T03:29:33.649 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509450 2024-09-15T03:29:33.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:29:33.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-15T03:29:33.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:29:33.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:29:33.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:29:34.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T03:29:34.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:29:34.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:29:34.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:29:34.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:29:34.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:29:34.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:29:34.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:29:34.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T03:29:34.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:29:34.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:29:34.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:29:35.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T03:29:35.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:29:35.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:29:35.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2024-09-15T03:29:35.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2024-09-15T03:29:35.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2024-09-15T03:29:35.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2024-09-15T03:29:35.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2024-09-15T03:29:35.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2024-09-15T03:29:35.616 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-09-15T03:29:35.616 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-09-15T03:29:35.616 INFO:tasks.workunit.client.0.smithi137.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.6616e-05 s, 72.3 MB/s 2024-09-15T03:29:35.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-15T03:29:35.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:35.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2024-09-15T03:29:35.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:35.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2024-09-15T03:29:35.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:35.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2024-09-15T03:29:36.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:36.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2024-09-15T03:29:36.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:36.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2024-09-15T03:29:36.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:36.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2024-09-15T03:29:36.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:36.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2024-09-15T03:29:36.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:36.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2024-09-15T03:29:36.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:36.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2024-09-15T03:29:36.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:36.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2024-09-15T03:29:37.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2024-09-15T03:29:37.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2024-09-15T03:29:37.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-15T03:29:37.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-15T03:29:37.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2024-09-15T03:29:37.218 INFO:tasks.workunit.client.0.smithi137.stdout:created pool test snap snap1 2024-09-15T03:29:37.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2024-09-15T03:29:37.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-15T03:29:37.220 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-09-15T03:29:37.220 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-09-15T03:29:37.220 INFO:tasks.workunit.client.0.smithi137.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.1554e-05 s, 79.5 MB/s 2024-09-15T03:29:37.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-15T03:29:37.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:37.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2024-09-15T03:29:37.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:37.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2024-09-15T03:29:37.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:37.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2024-09-15T03:29:37.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:37.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2024-09-15T03:29:38.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:38.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2024-09-15T03:29:38.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:38.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2024-09-15T03:29:38.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:38.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2024-09-15T03:29:38.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:38.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2024-09-15T03:29:38.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:38.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2024-09-15T03:29:38.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:38.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2024-09-15T03:29:38.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2024-09-15T03:29:38.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-15T03:29:38.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2024-09-15T03:29:39.011 INFO:tasks.workunit.client.0.smithi137.stdout:created pool test snap snap2 2024-09-15T03:29:39.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2024-09-15T03:29:39.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-15T03:29:39.014 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-09-15T03:29:39.014 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-09-15T03:29:39.014 INFO:tasks.workunit.client.0.smithi137.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.0812e-05 s, 80.6 MB/s 2024-09-15T03:29:39.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-15T03:29:39.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:39.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2024-09-15T03:29:39.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:39.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2024-09-15T03:29:39.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:39.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2024-09-15T03:29:39.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:39.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2024-09-15T03:29:39.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:39.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2024-09-15T03:29:39.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:39.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2024-09-15T03:29:40.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:40.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2024-09-15T03:29:40.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:40.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2024-09-15T03:29:40.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:40.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2024-09-15T03:29:40.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-15T03:29:40.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2024-09-15T03:29:40.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2024-09-15T03:29:40.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2024-09-15T03:29:40.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-15T03:29:40.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-15T03:29:40.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2024-09-15T03:29:40.690 INFO:tasks.workunit.client.0.smithi137.stdout:removed pool test snap snap1 2024-09-15T03:29:40.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-15T03:29:40.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:29:40.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:29:40.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:29:40.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:29:40.693 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:29:40.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:29:40.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:29:40.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:29:40.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:29:40.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:29:40.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:29:40.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:29:40.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:29:40.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:29:40.780 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:29:41.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:29:41.304 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T03:29:41.304 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T03:29:41.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:29:41.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:41.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:29:41.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836503 2024-09-15T03:29:41.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836503 2024-09-15T03:29:41.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503' 2024-09-15T03:29:41.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:41.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:29:41.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672980 2024-09-15T03:29:41.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672980 2024-09-15T03:29:41.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2024-09-15T03:29:41.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:41.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:29:41.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509454 2024-09-15T03:29:41.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509454 2024-09-15T03:29:41.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509454' 2024-09-15T03:29:41.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:29:41.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836503 2024-09-15T03:29:41.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:29:41.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:29:41.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836503 2024-09-15T03:29:41.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:29:41.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836503 2024-09-15T03:29:41.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-15T03:29:41.810 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836503 2024-09-15T03:29:41.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:29:42.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-15T03:29:42.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:29:43.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:29:43.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:29:43.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-15T03:29:43.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:29:43.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672980 2024-09-15T03:29:43.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:29:43.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:29:43.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672980 2024-09-15T03:29:43.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:29:43.525 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672980 2024-09-15T03:29:43.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672980 2024-09-15T03:29:43.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-15T03:29:43.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:29:43.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672980 2024-09-15T03:29:43.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:29:43.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509454 2024-09-15T03:29:43.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:29:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:29:43.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509454 2024-09-15T03:29:43.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:29:43.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509454 2024-09-15T03:29:43.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-15T03:29:43.873 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509454 2024-09-15T03:29:43.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:29:44.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-15T03:29:44.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:29:44.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:29:44.227 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:29:44.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T03:29:44.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:29:44.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:29:44.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:29:44.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:29:44.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:29:44.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:29:44.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:29:45.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T03:29:45.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:29:45.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:29:45.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:29:45.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T03:29:45.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:29:45.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:29:45.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-15T03:29:55.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-15T03:29:55.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-15T03:29:55.517 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-15T03:29:55.518 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-15T03:29:55.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:55.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-15T03:29:55.520 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-15T03:29:55.520 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:55.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-15T03:29:55.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-15T03:29:55.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:55.640 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-15T03:29:55.640 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:55.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.72194383399999995 2024-09-15T03:29:55.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.72194383399999995 2024-09-15T03:29:55.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:55.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-15T03:29:55.749 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-15T03:29:55.749 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:55.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-15T03:29:55.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-15T03:29:55.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:55.859 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-15T03:29:55.859 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:55.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .72194383399999995 + 0.62662463499999999 2024-09-15T03:29:55.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.34856846899999994 2024-09-15T03:29:55.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:55.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-15T03:29:55.968 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-15T03:29:55.968 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:56.075 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-15T03:29:56.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-15T03:29:56.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:56.077 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-15T03:29:56.077 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:56.185 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.34856846899999994 + 0.00041211500000000001 2024-09-15T03:29:56.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.34898058399999994001 2024-09-15T03:29:56.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:56.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-15T03:29:56.186 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-15T03:29:56.186 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:56.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-15T03:29:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-15T03:29:56.296 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:56.296 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-15T03:29:56.297 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:56.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.34898058399999994001 + 0.45306376399999998 2024-09-15T03:29:56.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.80204434799999992001 2024-09-15T03:29:56.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:56.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-15T03:29:56.406 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-15T03:29:56.406 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:56.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-15T03:29:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-15T03:29:56.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:56.516 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-15T03:29:56.516 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:56.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.80204434799999992001 + 0.26827721500000001 2024-09-15T03:29:56.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.07032156299999993001 2024-09-15T03:29:56.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:56.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-15T03:29:56.625 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-15T03:29:56.625 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:56.733 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-15T03:29:56.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-15T03:29:56.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:56.735 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-15T03:29:56.735 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:56.844 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.07032156299999993001 + 0.51839680300000002 2024-09-15T03:29:56.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.58871836599999995001 2024-09-15T03:29:56.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:56.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-15T03:29:56.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-15T03:29:56.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:56.953 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-15T03:29:56.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-15T03:29:56.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:56.955 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-15T03:29:56.955 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:57.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.58871836599999995001 + 0.72192473700000004 2024-09-15T03:29:57.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.31064310299999999001 2024-09-15T03:29:57.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:29:57.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:29:57.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-15T03:29:57.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-15T03:29:57.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:57.174 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-15T03:29:57.175 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:29:57.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.31064310299999999001 + 1.0849163639999999 2024-09-15T03:29:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.39555946699999989001 2024-09-15T03:29:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-15T03:29:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.39555946699999989001 > 0.0' 2024-09-15T03:29:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:29:57.284 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:29:57.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2024-09-15T03:29:57.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2024-09-15T03:29:57.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-15T03:29:57.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2024-09-15T03:29:57.500 INFO:tasks.workunit.client.0.smithi137.stdout:removed pool test snap snap2 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:29:57.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:29:57.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:29:57.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:29:57.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:29:57.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:29:57.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:29:57.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:29:58.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:29:58.049 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-09-15T03:29:58.049 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-09-15T03:29:58.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:29:58.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:58.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:29:58.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-15T03:29:58.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-15T03:29:58.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T03:29:58.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:58.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:29:58.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672986 2024-09-15T03:29:58.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672986 2024-09-15T03:29:58.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672986' 2024-09-15T03:29:58.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:29:58.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:29:58.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509459 2024-09-15T03:29:58.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509459 2024-09-15T03:29:58.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672986 2-64424509459' 2024-09-15T03:29:58.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:29:58.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-15T03:29:58.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:29:58.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:29:58.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-15T03:29:58.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:29:58.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-15T03:29:58.533 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836509 2024-09-15T03:29:58.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T03:29:58.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:29:58.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836509 2024-09-15T03:29:58.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:29:59.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:29:59.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:30:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T03:30:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:30:00.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672986 2024-09-15T03:30:00.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:30:00.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672986 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:30:00.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672986 2024-09-15T03:30:00.270 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672986 2024-09-15T03:30:00.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-15T03:30:00.270 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:30:00.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-15T03:30:00.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:30:00.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509459 2024-09-15T03:30:00.653 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:30:00.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:30:00.654 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509459 2024-09-15T03:30:00.654 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:30:00.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509459 2024-09-15T03:30:00.655 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509459 2024-09-15T03:30:00.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-15T03:30:00.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:30:01.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-15T03:30:01.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:30:01.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:30:01.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:30:01.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T03:30:01.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:30:01.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:30:01.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:30:01.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:30:01.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:30:01.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:30:01.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:30:01.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T03:30:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:30:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:30:01.816 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:30:02.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T03:30:02.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:30:02.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:30:02.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-15T03:30:12.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-15T03:30:12.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-15T03:30:12.279 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-15T03:30:12.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-15T03:30:12.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:12.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-15T03:30:12.282 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-15T03:30:12.282 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:12.401 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-15T03:30:12.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-15T03:30:12.403 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:12.404 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-15T03:30:12.404 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:12.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.80196130200000004 2024-09-15T03:30:12.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.80196130200000004 2024-09-15T03:30:12.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:12.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-15T03:30:12.514 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-15T03:30:12.515 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:12.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-15T03:30:12.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-15T03:30:12.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:12.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-15T03:30:12.624 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:12.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .80196130200000004 + 0.43716683000000001 2024-09-15T03:30:12.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.23912813200000005 2024-09-15T03:30:12.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:12.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-15T03:30:12.732 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-15T03:30:12.732 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:12.840 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-15T03:30:12.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-15T03:30:12.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:12.841 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-15T03:30:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:12.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.23912813200000005 + 0.10064743800000001 2024-09-15T03:30:12.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.33977557000000006 2024-09-15T03:30:12.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:12.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-15T03:30:12.953 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-15T03:30:12.953 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:13.060 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-15T03:30:13.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-15T03:30:13.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:13.062 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-15T03:30:13.062 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:13.171 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.33977557000000006 + 1.0164475959999999 2024-09-15T03:30:13.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.35622316599999996 2024-09-15T03:30:13.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:13.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-15T03:30:13.172 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-15T03:30:13.172 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:13.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-15T03:30:13.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-15T03:30:13.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:13.282 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-15T03:30:13.283 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:13.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.35622316599999996 + 0.36138211399999998 2024-09-15T03:30:13.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.71760527999999994 2024-09-15T03:30:13.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:13.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-15T03:30:13.392 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-15T03:30:13.392 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:13.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-15T03:30:13.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-15T03:30:13.502 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:13.502 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-15T03:30:13.502 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:13.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.71760527999999994 + 0.23999036600000001 2024-09-15T03:30:13.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.95759564599999995 2024-09-15T03:30:13.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:13.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-15T03:30:13.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-15T03:30:13.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:13.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.95759564599999995 + 0.59376204799999999 2024-09-15T03:30:13.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.55135769399999994 2024-09-15T03:30:13.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-15T03:30:13.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-15T03:30:13.829 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-15T03:30:13.830 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-15T03:30:13.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-15T03:30:13.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-15T03:30:13.939 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:13.939 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-15T03:30:14.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.55135769399999994 + 0.59362064599999997 2024-09-15T03:30:14.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.14497833999999991 2024-09-15T03:30:14.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-15T03:30:14.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.14497833999999991 > 0.0' 2024-09-15T03:30:14.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-15T03:30:14.049 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:30:14.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:30:14.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:30:14.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:30:14.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:30:14.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:30:14.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:30:14.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:30:14.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:30:14.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:30:14.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:30:14.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:30:14.161 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:30:14.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:30:14.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:30:14.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:30:14.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:30:14.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:30:14.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:30:14.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:30:14.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:30:14.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:30:14.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:30:14.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-15T03:30:14.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:30:14.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-snaptrim-stats 0 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:30:14.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T03:30:14.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-15T03:30:14.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:30:14.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173648 2024-09-15T03:30:14.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173648 2024-09-15T03:30:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:30:14.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:30:14.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T03:30:14.221 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:30:14.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:30:14.246 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2024-09-15T03:30:14.246 DEBUG:teuthology.orchestra.run.smithi137:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-09-15T03:30:14.308 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T03:30:14.314 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T03:30:14.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:30:14.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/ver-health 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2024-09-15T03:30:14.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none ' 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2024-09-15T03:30:14.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_check_version_health_2 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_check_version_health_3' 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:30:14.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:30:14.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:30:14.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:30:14.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:30:14.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:30:14.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:30:14.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:30:14.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:30:14.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:30:14.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:30:14.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:30:14.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:30:14.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:30:14.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:30:14.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:30:14.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:30:14.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:30:14.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:30:14.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:14.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:30:14.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:30:14.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:30:14.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-15T03:30:14.326 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:30:14.326 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.326 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:14.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-15T03:30:14.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:30:14.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:30:14.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:30:14.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:30:14.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:30:14.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:30:14.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:30:14.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:30:14.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:30:14.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:30:14.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:30:14.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:30:14.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:30:14.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:30:14.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:30:14.335 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:30:14.335 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.336 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:14.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:30:14.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:30:14.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:30:14.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-15T03:30:14.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:30:14.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.338 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:14.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T03:30:14.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:30:14.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:30:14.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-15T03:30:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-15T03:30:14.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:30:14.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:30:14.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get fsid 2024-09-15T03:30:15.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:30:15.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:30:15.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:30:15.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:30:15.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:30:15.033 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:30:15.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get mon_host 2024-09-15T03:30:15.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-15T03:30:15.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-15T03:30:15.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:30:15.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T03:30:15.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:30:15.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-15T03:30:15.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-15T03:30:15.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:30:15.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:30:15.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:30:15.708 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:30:15.708 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:15.708 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:15.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:30:15.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-15T03:30:16.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:30:16.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T03:30:16.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:30:16.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:30:16.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:30:16.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:30:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:30:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:30:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:30:16.100 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:30:16.100 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:16.100 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:16.100 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:30:16.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:30:16.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get fsid 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:30:16.174 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:30:16.175 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:16.175 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:16.175 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:30:16.175 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:30:16.175 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get mon_host 2024-09-15T03:30:16.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2024-09-15T03:30:16.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:30:16.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:30:16.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:30:16.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:30:16.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-15T03:30:16.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:30:16.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:30:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:30:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:30:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-15T03:30:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-15T03:30:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:30:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:30:16.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:30:16.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:30:16.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:30:16.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:30:16.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dbe73ab4-b884-4ef9-a6aa-0409d2efa96f 2024-09-15T03:30:16.260 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 dbe73ab4-b884-4ef9-a6aa-0409d2efa96f 2024-09-15T03:30:16.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dbe73ab4-b884-4ef9-a6aa-0409d2efa96f' 2024-09-15T03:30:16.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:30:16.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIVOZmpIy3DxAAAmXBXu3oS9MdzJA54wCp1A== 2024-09-15T03:30:16.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIVOZmpIy3DxAAAmXBXu3oS9MdzJA54wCp1A=="}' 2024-09-15T03:30:16.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dbe73ab4-b884-4ef9-a6aa-0409d2efa96f -i td/ver-health/0/new.json 2024-09-15T03:30:22.564 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:30:22.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-15T03:30:22.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIVOZmpIy3DxAAAmXBXu3oS9MdzJA54wCp1A== --osd-uuid dbe73ab4-b884-4ef9-a6aa-0409d2efa96f 2024-09-15T03:30:22.599 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:22.597+0000 7f086731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:22.601 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:22.599+0000 7f086731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:22.605 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:22.600+0000 7f086731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:22.605 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:22.600+0000 7f086731f440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-15T03:30:22.605 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:22.600+0000 7f086731f440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-15T03:30:22.605 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:22.600+0000 7f086731f440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-15T03:30:22.699 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:22.697+0000 7f086731f440 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-15T03:30:28.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-15T03:30:28.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:30:28.037 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:30:28.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:30:28.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:30:28.229 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:30:28.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:30:28.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:30:28.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:30:28.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:30:28.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:30:28.251 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:28.250+0000 7f89f3215440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:28.262 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:28.261+0000 7f89f3215440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:28.263 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:28.262+0000 7f89f3215440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:28.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:30:28.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:30:28.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:30:28.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:30:28.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:30:28.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:28.413 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:30:28.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:30:28.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:28.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:28.793 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:28.792+0000 7f89f3215440 -1 Falling back to public interface 2024-09-15T03:30:29.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:29.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:29.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:30:29.596 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:30:29.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:29.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:29.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:30.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:30.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:30.781 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:30:30.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:30:30.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:30.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:30.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:31.409 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:31.408+0000 7f89f3215440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:30:31.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:31.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:31.968 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:30:31.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:30:31.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:31.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:32.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:33.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:33.399 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:30:33.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:30:33.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:33.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:33.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:34.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:34.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:34.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:30:34.587 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:30:34.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:34.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:34.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:35.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:35.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:35.775 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:30:35.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:30:35.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:35.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:35.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:36.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:36.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:36.961 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:30:36.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:30:36.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:36.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:37.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:38.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:38.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:38.147 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:30:38.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:30:38.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:38.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:38.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:39.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:39.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:39.334 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:30:39.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:30:39.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:39.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:39.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:40.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:40.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:40.521 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:30:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:30:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:40.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:41.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:41.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:41.708 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:30:41.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:30:41.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:41.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:41.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:42.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:42.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:42.893 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:30:42.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:30:42.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:42.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:43.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:44.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:44.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:44.078 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:30:44.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:30:44.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:44.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:44.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:45.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:45.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:45.265 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:30:45.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:30:45.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:45.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:45.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:46.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:46.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:46.449 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:30:46.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:30:46.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:46.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:46.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:47.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:47.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:47.636 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:30:47.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:30:47.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:47.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:47.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:48.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:48.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:48.827 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:30:48.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:30:48.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:48.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:50.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:50.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:50.010 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:30:50.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:30:50.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:50.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:50.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:51.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:51.188 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:30:51.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:51.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:30:51.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:51.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:51.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:52.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:52.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:52.377 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:30:52.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:30:52.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:52.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:52.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:53.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:53.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:53.563 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-09-15T03:30:53.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-15T03:30:53.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:53.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:53.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:30:54.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:30:54.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:30:54.751 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-09-15T03:30:54.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-15T03:30:54.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:30:54.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:30:55.018 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/190245,v1:127.0.0.1:6801/190245] [v2:127.0.0.1:6802/190245,v1:127.0.0.1:6803/190245] exists,up dbe73ab4-b884-4ef9-a6aa-0409d2efa96f 2024-09-15T03:30:55.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:30:55.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:30:55.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:30:55.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2024-09-15T03:30:55.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:30:55.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:30:55.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:30:55.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:30:55.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5b2a6779-5fca-448e-a219-27a645e55562 2024-09-15T03:30:55.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5b2a6779-5fca-448e-a219-27a645e55562' 2024-09-15T03:30:55.026 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 5b2a6779-5fca-448e-a219-27a645e55562 2024-09-15T03:30:55.026 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:30:55.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvVOZmH8QNAhAA6yQol0tYdZhnLGfsrpGluw== 2024-09-15T03:30:55.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvVOZmH8QNAhAA6yQol0tYdZhnLGfsrpGluw=="}' 2024-09-15T03:30:55.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5b2a6779-5fca-448e-a219-27a645e55562 -i td/ver-health/1/new.json 2024-09-15T03:30:55.520 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:30:55.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-15T03:30:55.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvVOZmH8QNAhAA6yQol0tYdZhnLGfsrpGluw== --osd-uuid 5b2a6779-5fca-448e-a219-27a645e55562 2024-09-15T03:30:55.553 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:55.552+0000 7f8ce6475440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:55.555 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:55.554+0000 7f8ce6475440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:55.556 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:55.555+0000 7f8ce6475440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:30:55.557 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:55.555+0000 7f8ce6475440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-15T03:30:55.557 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:55.555+0000 7f8ce6475440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-15T03:30:55.557 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:55.555+0000 7f8ce6475440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-15T03:30:55.670 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:30:55.668+0000 7f8ce6475440 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-15T03:31:01.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-15T03:31:01.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:31:01.199 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:31:01.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:31:01.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:31:01.393 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:31:01.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:31:01.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:31:01.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:31:01.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:31:01.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:31:01.414 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:01.412+0000 7fb03134c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:01.424 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:01.423+0000 7fb03134c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:01.425 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:01.424+0000 7fb03134c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:01.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:31:01.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:31:01.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:31:01.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:31:01.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:31:01.655 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:31:01.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:01.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:31:01.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:01.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:01.963 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:01.962+0000 7fb03134c440 -1 Falling back to public interface 2024-09-15T03:31:02.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:02.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:02.929 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:31:02.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:31:02.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:02.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:03.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:04.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:04.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:04.114 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:31:04.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:31:04.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:04.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:04.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:04.631 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:04.630+0000 7fb03134c440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:05.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:06.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:06.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:06.615 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:31:06.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:31:06.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:06.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:06.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:07.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:07.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:07.800 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:31:07.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:31:07.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:07.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:07.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:08.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:08.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:08.987 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:31:08.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:31:08.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:08.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:09.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:10.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:10.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:10.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:31:10.175 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:31:10.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:10.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:10.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:11.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:11.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:11.361 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:31:11.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:31:11.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:11.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:11.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:12.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:12.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:12.547 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:31:12.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:31:12.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:12.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:12.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:13.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:13.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:13.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:31:13.733 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:31:13.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:13.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:13.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:14.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:14.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:14.920 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:31:14.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:31:14.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:14.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:15.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:16.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:16.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:16.107 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:31:16.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:31:16.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:16.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:16.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:17.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:17.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:17.295 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:31:17.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:31:17.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:17.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:17.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:18.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:18.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:18.479 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:31:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:31:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:18.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:18.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:19.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:19.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:19.666 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:31:19.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:31:19.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:19.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:19.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:20.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:20.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:20.849 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:31:20.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:31:20.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:20.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:21.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:22.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:22.031 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:31:22.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:31:22.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:22.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:22.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:23.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:23.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:23.217 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:31:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:31:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:23.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:24.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:24.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:24.402 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:31:24.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:31:24.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:24.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:24.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:25.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:25.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:25.585 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:31:25.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:31:25.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:25.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:25.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:26.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:26.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:26.768 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-09-15T03:31:26.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-15T03:31:26.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:26.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:27.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:28.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:28.010 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-09-15T03:31:28.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:28.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-15T03:31:28.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:28.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:31:28.227 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/192126,v1:127.0.0.1:6809/192126] [v2:127.0.0.1:6810/192126,v1:127.0.0.1:6811/192126] exists,up 5b2a6779-5fca-448e-a219-27a645e55562 2024-09-15T03:31:28.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:31:28.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:31:28.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-15T03:31:28.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:31:28.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:31:28.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-15T03:31:28.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:31:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0e4cdec-8690-4bbc-8903-7bbdee2591d6 2024-09-15T03:31:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d0e4cdec-8690-4bbc-8903-7bbdee2591d6' 2024-09-15T03:31:28.235 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 d0e4cdec-8690-4bbc-8903-7bbdee2591d6 2024-09-15T03:31:28.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:31:28.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQVeZmxg2TDhAA9vbGXnVggydOPDY7RMlrrw== 2024-09-15T03:31:28.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQVeZmxg2TDhAA9vbGXnVggydOPDY7RMlrrw=="}' 2024-09-15T03:31:28.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0e4cdec-8690-4bbc-8903-7bbdee2591d6 -i td/ver-health/2/new.json 2024-09-15T03:31:28.900 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:31:28.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-15T03:31:28.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQVeZmxg2TDhAA9vbGXnVggydOPDY7RMlrrw== --osd-uuid d0e4cdec-8690-4bbc-8903-7bbdee2591d6 2024-09-15T03:31:28.934 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:28.933+0000 7f2c3171d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:28.985 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:28.985+0000 7f2c3171d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:28.986 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:28.986+0000 7f2c3171d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:28.987 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:28.986+0000 7f2c3171d440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-15T03:31:28.987 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:28.986+0000 7f2c3171d440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-15T03:31:28.987 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:28.986+0000 7f2c3171d440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-15T03:31:29.059 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:29.059+0000 7f2c3171d440 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-15T03:31:34.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-15T03:31:34.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:31:34.623 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:31:34.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:31:34.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:31:34.818 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:31:34.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:31:34.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:31:34.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:31:34.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:31:34.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:31:34.841 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:34.840+0000 7f0ece523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:34.855 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:34.854+0000 7f0ece523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:34.856 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:34.855+0000 7f0ece523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:31:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:31:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:31:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:31:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:31:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:31:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:31:35.061 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:31:35.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:35.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:35.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:35.387 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:35.387+0000 7f0ece523440 -1 Falling back to public interface 2024-09-15T03:31:36.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:36.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:36.254 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:31:36.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:31:36.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:36.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:36.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:37.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:37.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:37.435 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:31:37.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:31:37.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:37.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:37.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:38.044 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:31:38.044+0000 7f0ece523440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:31:38.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:38.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:38.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:31:38.621 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:31:38.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:38.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:38.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:39.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:39.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:39.806 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:31:39.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:31:39.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:39.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:39.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:40.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:40.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:40.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:31:40.994 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:31:40.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:40.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:41.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:42.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:42.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:42.179 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:31:42.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:31:42.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:42.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:42.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:43.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:43.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:43.366 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:31:43.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:31:43.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:43.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:43.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:44.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:44.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:44.551 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:31:44.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:31:44.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:44.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:44.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:45.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:45.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:45.734 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:31:45.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:31:45.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:45.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:45.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:46.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:46.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:46.919 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:31:46.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:31:46.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:46.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:47.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:48.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:48.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:48.106 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:31:48.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:31:48.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:48.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:48.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:49.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:49.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:49.292 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:31:49.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:31:49.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:49.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:49.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:50.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:50.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:50.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:31:50.480 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:31:50.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:50.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:50.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:51.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:51.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:51.662 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:31:51.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:31:51.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:51.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:51.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:52.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:52.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:52.846 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:31:52.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:31:52.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:52.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:53.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:54.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:54.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:54.033 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:31:54.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:31:54.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:54.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:54.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:55.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:55.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:55.220 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:31:55.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:31:55.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:55.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:55.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:56.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:56.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:56.406 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:31:56.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:31:56.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:56.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:56.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:57.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:57.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:57.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:31:57.592 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:31:57.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:57.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:57.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:58.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:58.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:58.779 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:31:58.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:31:58.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:58.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:31:58.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:31:59.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:31:59.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:31:59.956 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-09-15T03:31:59.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-15T03:31:59.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:31:59.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:32:00.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:01.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:01.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:01.138 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-09-15T03:32:01.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-15T03:32:01.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:01.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:32:01.441 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/194006,v1:127.0.0.1:6817/194006] [v2:127.0.0.1:6818/194006,v1:127.0.0.1:6819/194006] exists,up d0e4cdec-8690-4bbc-8903-7bbdee2591d6 2024-09-15T03:32:01.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:32:01.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:32:01.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:32:01.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2024-09-15T03:32:06.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:06.622 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_OK 2024-09-15T03:32:06.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:06.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2024-09-15T03:32:06.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2024-09-15T03:32:06.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:32:06.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:32:06.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:32:06.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:32:06.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:32:06.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:32:06.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-15T03:32:06.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:32:06.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-15T03:32:06.982 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:32:06.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:32:06.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:32:06.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-15T03:32:06.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:32:06.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:32:06.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:32:06.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:32:07.001 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:07.000+0000 7f690c00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:07.074 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:07.072+0000 7f690c00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:07.075 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:07.074+0000 7f690c00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:07.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:32:07.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:32:07.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:32:07.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:32:07.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:32:07.243 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:32:07.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:07.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:32:07.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:07.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:32:07.590 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:07.589+0000 7f690c00d440 -1 Falling back to public interface 2024-09-15T03:32:07.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:08.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:08.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:08.639 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:32:08.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:32:08.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:08.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:32:08.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:09.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:09.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:09.824 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:32:09.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:32:09.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:09.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:32:09.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:10.535 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:10.534+0000 7f690c00d440 -1 osd.1 16 log_to_monitors true 2024-09-15T03:32:10.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:10.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:10.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:32:10.999 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:32:10.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:10.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:32:11.252 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6808/195133,v1:127.0.0.1:6809/195133] [v2:127.0.0.1:6810/195133,v1:127.0.0.1:6811/195133] exists,up 5b2a6779-5fca-448e-a219-27a645e55562 2024-09-15T03:32:11.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:11.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:11.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:12.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:12.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:12.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:12.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:12.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:13.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:14.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:14.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:14.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:14.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:15.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:16.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:16.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:16.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:16.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:16.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:17.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:17.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:17.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:17.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:17.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:18.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:18.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:18.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:18.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:18.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:19.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:19.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:19.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:19.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:19.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:20.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:20.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:20.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:20.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:21.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-15T03:32:21.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-15T03:32:21.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-15T03:32:21.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-15T03:32:21.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-15T03:32:21.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:21.386 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_WARN There is a daemon running an older version of ceph; no active mgr 2024-09-15T03:32:21.386 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-15T03:32:21.386 INFO:tasks.workunit.client.0.smithi137.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-15T03:32:21.386 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-15T03:32:21.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:21.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:32:21.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:21.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-15T03:32:21.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:21.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-15T03:32:22.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2024-09-15T03:32:22.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:32:22.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:32:22.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:32:22.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:32:22.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:32:22.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:32:22.175 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:32:22.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:32:22.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:32:22.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:32:22.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-15T03:32:22.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:32:22.177 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:32:22.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:32:22.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-15T03:32:22.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:32:22.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:32:22.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:32:22.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:32:22.196 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:22.194+0000 7f9528013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:22.197 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:22.196+0000 7f9528013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:22.199 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:22.197+0000 7f9528013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:22.720 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:22.718+0000 7f9528013440 -1 Falling back to public interface 2024-09-15T03:32:22.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:32:22.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:32:22.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:32:22.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:32:22.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:32:22.909 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:32:22.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:22.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:32:22.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:22.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:32:23.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:24.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:24.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:24.139 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:32:24.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:32:24.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:24.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:32:24.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:25.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:25.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:25.325 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:32:25.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:32:25.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:25.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:32:25.495 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:25.494+0000 7f9528013440 -1 osd.2 19 log_to_monitors true 2024-09-15T03:32:25.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:26.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:26.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:26.508 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:32:26.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:32:26.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:26.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:32:26.745 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6816/196022,v1:127.0.0.1:6817/196022] [v2:127.0.0.1:6818/196022,v1:127.0.0.1:6819/196022] exists,up d0e4cdec-8690-4bbc-8903-7bbdee2591d6 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:32:26.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:32:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:32:26.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:32:26.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:32:26.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:32:26.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:32:26.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:32:26.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:32:26.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-15T03:32:26.857 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:32:26.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:32:26.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:32:26.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-15T03:32:26.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:32:26.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:32:26.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:32:26.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:32:26.875 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:26.875+0000 7faf46875440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:26.946 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:26.945+0000 7faf46875440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:26.947+0000 7faf46875440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:27.463 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:27.463+0000 7faf46875440 -1 Falling back to public interface 2024-09-15T03:32:27.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:32:27.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:32:27.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:32:27.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:32:27.485 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:32:27.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:32:27.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:27.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:32:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:27.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:27.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:28.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:28.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:28.699 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:32:28.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:32:28.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:28.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:28.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:29.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:29.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:29.889 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:32:29.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:32:29.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:29.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:30.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:30.325 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:30.324+0000 7faf46875440 -1 osd.0 22 log_to_monitors true 2024-09-15T03:32:31.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:31.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:31.064 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:32:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:32:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:31.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:31.326 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/196626,v1:127.0.0.1:6801/196626] [v2:127.0.0.1:6802/196626,v1:127.0.0.1:6803/196626] exists,up dbe73ab4-b884-4ef9-a6aa-0409d2efa96f 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-15T03:32:31.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-15T03:32:31.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-15T03:32:31.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:31.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:31.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:31.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:32.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:32.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:32.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:32.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:32.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:33.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:33.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:33.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:33.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:34.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:35.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:35.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:35.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:35.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:35.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:32:36.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:32:36.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:32:36.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:32:36.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:36.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-15T03:32:36.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-15T03:32:36.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-15T03:32:36.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-15T03:32:36.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-15T03:32:36.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:36.700 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph; no active mgr 2024-09-15T03:32:36.700 INFO:tasks.workunit.client.0.smithi137.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-15T03:32:36.700 INFO:tasks.workunit.client.0.smithi137.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-15T03:32:36.700 INFO:tasks.workunit.client.0.smithi137.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-15T03:32:36.700 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-15T03:32:36.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:36.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:32:37.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:37.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-15T03:32:37.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:37.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-15T03:32:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2024-09-15T03:32:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-15T03:32:37.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-15T03:32:37.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:32:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:32:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:32:37.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:32:37.720 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:32:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:32:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:32:37.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:32:37.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:32:37.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:32:37.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:32:37.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:32:37.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:32:37.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:32:37.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:32:37.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:37.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:32:37.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:32:37.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:37.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:32:37.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:32:37.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:32:37.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:32:37.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:37.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:37.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:32:37.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:32:37.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:32:37.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:32:37.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:32:37.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:32:37.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:32:37.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:32:37.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:32:37.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:32:37.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:32:37.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:32:37.853 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:32:37.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:37.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:32:37.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:32:37.854 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:37.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:32:37.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:32:37.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:32:37.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:32:37.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:37.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:37.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:32:37.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:32:37.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:32:37.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-15T03:32:37.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:32:37.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:37.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:37.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T03:32:37.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:32:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:32:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:32:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-15T03:32:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2024-09-15T03:32:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:32:37.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:32:37.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:32:37.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:32:37.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:32:37.864 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:32:37.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:32:37.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:32:37.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:32:37.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:32:37.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:32:37.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:37.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:32:37.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:32:37.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:37.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:32:37.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:32:37.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:32:37.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:32:37.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:37.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:37.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:32:37.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:32:37.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:32:37.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-15T03:32:37.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:32:37.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:37.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:37.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T03:32:37.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:32:37.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:32:37.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:32:37.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-15T03:32:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-15T03:32:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-15T03:32:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:32:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:32:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:32:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-15T03:32:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-15T03:32:38.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:32:38.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:32:38.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:32:38.257 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:32:38.257 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:38.257 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:38.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:38.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-15T03:32:38.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:32:38.558 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:32:38.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:32:38.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:32:38.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:32:38.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:32:38.559 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:32:38.559 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:32:38.559 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:32:38.560 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:32:38.560 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:38.560 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:38.560 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T03:32:38.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:32:38.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get fsid 2024-09-15T03:32:38.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:32:38.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get mon_host 2024-09-15T03:32:38.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-15T03:32:38.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-15T03:32:38.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:32:38.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T03:32:38.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:32:38.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-15T03:32:38.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:39.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-15T03:32:39.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:32:39.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T03:32:39.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:32:39.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:32:39.708 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:32:39.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get fsid 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:39.794 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:39.795 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:32:39.795 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:32:39.795 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get mon_host 2024-09-15T03:32:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2024-09-15T03:32:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:32:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:32:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:32:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:32:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-15T03:32:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:32:39.873 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:32:39.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:32:39.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-15T03:32:39.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:32:39.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c430bee4-4b33-4c19-a688-8e9c774c41b8 2024-09-15T03:32:39.875 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 c430bee4-4b33-4c19-a688-8e9c774c41b8 2024-09-15T03:32:39.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c430bee4-4b33-4c19-a688-8e9c774c41b8' 2024-09-15T03:32:39.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:32:39.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXVeZmGeXINBAA2107I7yuqPyIpfHmRz3elw== 2024-09-15T03:32:39.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXVeZmGeXINBAA2107I7yuqPyIpfHmRz3elw=="}' 2024-09-15T03:32:39.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c430bee4-4b33-4c19-a688-8e9c774c41b8 -i td/ver-health/0/new.json 2024-09-15T03:32:46.179 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:32:46.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-15T03:32:46.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXVeZmGeXINBAA2107I7yuqPyIpfHmRz3elw== --osd-uuid c430bee4-4b33-4c19-a688-8e9c774c41b8 2024-09-15T03:32:46.214 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:46.212+0000 7f85ada30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:46.215 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:46.214+0000 7f85ada30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:46.217 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:46.215+0000 7f85ada30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:46.217 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:46.215+0000 7f85ada30440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-15T03:32:46.217 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:46.215+0000 7f85ada30440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-15T03:32:46.218 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:46.215+0000 7f85ada30440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-15T03:32:46.305 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:46.297+0000 7f85ada30440 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-15T03:32:51.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-15T03:32:51.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:32:51.849 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:32:51.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:32:51.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:32:52.041 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:32:52.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:32:52.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:32:52.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:32:52.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:32:52.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:52.062+0000 7f15c1357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:52.077 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:52.076+0000 7f15c1357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:52.078 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:52.077+0000 7f15c1357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:32:52.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:32:52.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:32:52.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:32:52.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:32:52.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:32:52.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:52.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:32:52.241 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:32:52.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:52.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:52.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:52.606 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:52.605+0000 7f15c1357440 -1 Falling back to public interface 2024-09-15T03:32:53.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:53.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:53.426 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:32:53.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:32:53.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:53.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:53.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:54.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:54.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:54.609 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:32:54.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:32:54.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:54.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:54.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:55.207 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:32:55.205+0000 7f15c1357440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:32:55.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:55.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:55.790 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:32:55.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:32:55.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:55.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:56.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:57.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:57.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:57.098 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:32:57.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:32:57.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:57.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:57.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:58.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:58.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:58.280 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:32:58.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:32:58.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:58.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:58.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:32:59.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:32:59.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:32:59.466 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:32:59.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:32:59.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:32:59.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:32:59.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:00.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:00.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:00.652 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:33:00.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:33:00.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:00.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:00.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:01.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:01.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:01.836 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:33:01.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:33:01.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:01.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:02.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:03.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:03.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:03.023 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:33:03.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:33:03.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:03.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:03.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:04.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:04.210 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:33:04.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:04.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:33:04.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:04.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:04.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:05.396 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:33:05.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:33:05.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:05.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:05.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:06.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:06.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:06.583 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:33:06.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:33:06.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:06.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:06.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:07.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:07.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:07.771 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:33:07.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:33:07.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:07.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:07.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:08.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:08.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:08.961 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:33:08.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:33:08.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:08.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:09.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:10.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:10.143 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:33:10.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:33:10.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:10.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:10.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:11.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:11.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:11.329 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:33:11.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:33:11.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:11.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:11.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:12.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:12.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:12.516 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:33:12.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:33:12.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:12.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:12.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:13.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:13.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:13.701 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:33:13.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:33:13.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:13.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:14.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:14.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:14.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:33:14.890 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:33:14.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:14.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:15.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:16.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:16.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:16.075 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:33:16.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:33:16.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:16.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:16.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:17.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:17.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:17.258 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-09-15T03:33:17.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-15T03:33:17.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:17.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:17.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:18.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:18.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:18.446 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-09-15T03:33:18.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-15T03:33:18.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:18.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:33:18.643 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/198480,v1:127.0.0.1:6801/198480] [v2:127.0.0.1:6802/198480,v1:127.0.0.1:6803/198480] exists,up c430bee4-4b33-4c19-a688-8e9c774c41b8 2024-09-15T03:33:18.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:33:18.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:33:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-15T03:33:18.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-15T03:33:18.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:33:18.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:33:18.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:33:18.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:33:18.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:33:18.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:33:18.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:33:18.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:33:18.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:33:18.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:33:18.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-15T03:33:18.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:33:18.650 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 911c5865-daa8-4077-81f4-f7e5205194f6 2024-09-15T03:33:18.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=911c5865-daa8-4077-81f4-f7e5205194f6 2024-09-15T03:33:18.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 911c5865-daa8-4077-81f4-f7e5205194f6' 2024-09-15T03:33:18.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:33:18.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+VeZmEw1PJxAA41aym7wZ8Cr5pavG1PUETA== 2024-09-15T03:33:18.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+VeZmEw1PJxAA41aym7wZ8Cr5pavG1PUETA=="}' 2024-09-15T03:33:18.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 911c5865-daa8-4077-81f4-f7e5205194f6 -i td/ver-health/1/new.json 2024-09-15T03:33:19.367 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:33:19.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-15T03:33:19.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+VeZmEw1PJxAA41aym7wZ8Cr5pavG1PUETA== --osd-uuid 911c5865-daa8-4077-81f4-f7e5205194f6 2024-09-15T03:33:19.402 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:19.401+0000 7f35d1d2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:19.462 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:19.460+0000 7f35d1d2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:19.461+0000 7f35d1d2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:19.462+0000 7f35d1d2b440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-15T03:33:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:19.462+0000 7f35d1d2b440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-15T03:33:19.464 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:19.462+0000 7f35d1d2b440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-15T03:33:19.552 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:19.551+0000 7f35d1d2b440 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-15T03:33:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-15T03:33:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:33:25.166 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:33:25.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:33:25.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:33:25.358 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:33:25.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:33:25.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:33:25.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:33:25.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:33:25.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:33:25.380 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:25.379+0000 7fab6701f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:25.397 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:25.396+0000 7fab6701f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:25.398 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:25.397+0000 7fab6701f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:25.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:33:25.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:33:25.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:33:25.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:33:25.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:33:25.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:25.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:33:25.592 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:33:25.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:25.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:25.927 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:25.927+0000 7fab6701f440 -1 Falling back to public interface 2024-09-15T03:33:26.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:27.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:27.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:27.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:33:27.224 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:33:27.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:27.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:27.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:28.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:28.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:28.273 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:33:28.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:33:28.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:28.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:28.394 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:28.394+0000 7fab6701f440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:33:28.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:29.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:29.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:29.456 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:33:29.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:33:29.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:29.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:29.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:30.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:30.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:30.769 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:33:30.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:33:30.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:30.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:30.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:31.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:31.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:31.954 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:33:31.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:33:31.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:31.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:32.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:33.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:33.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:33.144 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:33:33.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:33:33.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:33.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:33.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:34.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:34.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:34.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:33:34.332 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:33:34.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:34.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:34.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:35.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:35.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:35.518 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:33:35.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:33:35.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:35.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:35.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:36.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:36.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:36.705 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:33:36.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:33:36.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:36.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:36.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:37.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:37.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:37.893 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:33:37.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:33:37.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:37.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:38.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:39.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:39.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:39.080 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:33:39.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:33:39.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:39.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:39.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:40.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:40.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:40.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:33:40.266 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:33:40.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:40.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:40.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:41.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:41.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:41.451 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:33:41.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:33:41.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:41.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:41.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:42.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:42.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:42.635 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:33:42.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:33:42.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:42.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:42.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:43.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:43.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:43.823 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:33:43.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:33:43.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:43.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:44.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:45.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:45.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:45.009 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:33:45.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:33:45.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:45.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:45.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:46.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:46.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:46.354 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:33:46.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:33:46.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:46.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:46.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:47.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:47.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:47.540 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:33:47.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:33:47.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:47.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:48.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:48.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:48.728 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:33:48.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:33:48.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:48.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:48.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:49.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:49.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:49.914 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:33:49.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:33:49.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:49.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:50.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:51.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:51.101 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-09-15T03:33:51.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:51.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-15T03:33:51.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:51.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:33:51.286 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/200360,v1:127.0.0.1:6809/200360] [v2:127.0.0.1:6810/200360,v1:127.0.0.1:6811/200360] exists,up 911c5865-daa8-4077-81f4-f7e5205194f6 2024-09-15T03:33:51.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:33:51.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:33:51.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-15T03:33:51.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:33:51.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-15T03:33:51.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:33:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d3876208-76ba-4131-8ba0-b7aefffb1653 2024-09-15T03:33:51.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d3876208-76ba-4131-8ba0-b7aefffb1653' 2024-09-15T03:33:51.294 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 d3876208-76ba-4131-8ba0-b7aefffb1653 2024-09-15T03:33:51.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:33:51.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfVeZm0c0MEhAAc0Kr2pmF3gfciDfpT4dVLQ== 2024-09-15T03:33:51.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfVeZm0c0MEhAAc0Kr2pmF3gfciDfpT4dVLQ=="}' 2024-09-15T03:33:51.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d3876208-76ba-4131-8ba0-b7aefffb1653 -i td/ver-health/2/new.json 2024-09-15T03:33:51.667 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:33:51.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-15T03:33:51.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfVeZm0c0MEhAAc0Kr2pmF3gfciDfpT4dVLQ== --osd-uuid d3876208-76ba-4131-8ba0-b7aefffb1653 2024-09-15T03:33:51.700 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:51.699+0000 7f763980c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:51.702 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:51.701+0000 7f763980c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:51.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:51.702+0000 7f763980c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:51.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:51.702+0000 7f763980c440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-15T03:33:51.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:51.702+0000 7f763980c440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-15T03:33:51.704 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:51.702+0000 7f763980c440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-15T03:33:51.776 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:51.775+0000 7f763980c440 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-15T03:33:57.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-15T03:33:57.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:33:57.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:33:57.516 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:33:57.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:33:57.708 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:33:57.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:33:57.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:33:57.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:33:57.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:33:57.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:33:57.730 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:57.729+0000 7f5ffe214440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:57.740 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:57.739+0000 7f5ffe214440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:57.741 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:57.740+0000 7f5ffe214440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:33:57.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:33:57.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:33:57.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:33:57.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:33:57.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:33:57.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:57.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:33:57.927 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:33:57.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:57.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:33:58.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:33:58.273 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:33:58.272+0000 7f5ffe214440 -1 Falling back to public interface 2024-09-15T03:33:59.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:33:59.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:33:59.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:33:59.138 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:33:59.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:33:59.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:33:59.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:00.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:00.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:00.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:34:00.319 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:34:00.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:00.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:00.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:00.839 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:00.837+0000 7f5ffe214440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:34:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:01.505 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:34:01.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:34:01.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:01.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:01.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:02.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:02.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:02.826 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-09-15T03:34:02.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:34:02.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:02.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:03.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:04.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:04.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:04.012 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-09-15T03:34:04.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:34:04.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:04.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:04.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:05.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:05.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:05.199 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-09-15T03:34:05.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:34:05.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:05.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:05.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:06.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:06.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:06.381 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-09-15T03:34:06.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:34:06.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:06.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:06.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:07.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:07.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:07.566 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-09-15T03:34:07.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:34:07.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:07.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:07.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:08.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:08.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:08.751 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-09-15T03:34:08.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:34:08.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:08.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:08.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:09.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:09.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:09.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:34:09.937 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-09-15T03:34:09.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:09.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:10.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:11.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:11.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:11.124 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-09-15T03:34:11.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:34:11.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:11.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:11.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:12.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:12.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:12.309 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-09-15T03:34:12.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:34:12.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:12.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:12.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:13.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:13.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:13.493 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-09-15T03:34:13.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:34:13.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:13.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:13.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:14.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:14.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:14.677 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-09-15T03:34:14.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:34:14.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:14.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:14.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:15.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:15.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:15.855 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-09-15T03:34:15.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:34:15.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:15.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:16.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:17.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:17.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:17.042 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-09-15T03:34:17.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:34:17.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:17.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:17.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:18.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:18.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:18.226 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-09-15T03:34:18.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:34:18.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:18.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:18.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:19.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:19.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:19.405 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-09-15T03:34:19.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:34:19.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:19.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:20.587 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-09-15T03:34:20.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T03:34:20.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:20.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:20.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:21.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:21.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:21.776 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-09-15T03:34:21.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-15T03:34:21.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:21.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:22.353 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/202214,v1:127.0.0.1:6817/202214] [v2:127.0.0.1:6818/202214,v1:127.0.0.1:6819/202214] exists,up d3876208-76ba-4131-8ba0-b7aefffb1653 2024-09-15T03:34:22.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:34:22.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:34:22.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:34:22.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2024-09-15T03:34:22.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-15T03:34:22.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:34:22.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:34:22.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:34:22.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2024-09-15T03:34:22.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:22.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:34:22.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:34:22.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2024-09-15T03:34:22.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-15T03:34:22.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:34:22.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2024-09-15T03:34:22.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:34:22.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2024-09-15T03:34:22.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:34:23.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:34:23.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:23.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:23.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:23.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:23.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:23.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:23.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:34:23.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:34:23.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-15T03:34:23.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:23.082 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:23.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:23.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-15T03:34:23.098 INFO:tasks.workunit.client.0.smithi137.stdout:starting mds.m at 2024-09-15T03:34:23.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2024-09-15T03:34:23.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-15T03:34:23.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-15T03:34:23.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2024-09-15T03:34:23.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-15T03:34:23.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2024-09-15T03:34:23.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-15T03:34:23.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:23.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:23.104 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:23.104 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:23.104 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:23.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:23.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-15T03:34:23.126 INFO:tasks.workunit.client.0.smithi137.stdout:starting mds.n at 2024-09-15T03:34:23.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2024-09-15T03:34:28.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:28.642 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_OK 2024-09-15T03:34:28.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:28.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2024-09-15T03:34:29.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2024-09-15T03:34:29.182 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:34:29.182 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:34:29.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:34:29.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:34:29.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:34:29.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-15T03:34:29.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2024-09-15T03:34:29.304 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:29.304+0000 7f5fb9acc900 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2024-09-15T03:34:29.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:34:29.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:29.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:29.307 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:29.307 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:29.307 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:29.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:29.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2024-09-15T03:34:29.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:34:29.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T03:34:29.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:34:29.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:34:29.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:34:29.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:34:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:34:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:34:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:34:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:34:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:29.549 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:34:29.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:34:29.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get fsid 2024-09-15T03:34:29.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T03:34:29.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:34:29.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:34:29.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:34:29.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get mon_host 2024-09-15T03:34:29.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2024-09-15T03:34:29.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:34:29.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:34:29.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:34:29.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:34:29.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:34:29.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:34:29.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-15T03:34:29.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-15T03:34:29.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-15T03:34:29.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:29.804 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:29.805 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:29.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-15T03:34:29.877 INFO:tasks.workunit.client.0.smithi137.stdout:starting mds.m at 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:34:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-15T03:34:30.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:34:31.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:34:31.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:34:31.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:34:31.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-15T03:34:31.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:34:32.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:34:32.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:34:32.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:34:32.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-15T03:34:33.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:34:34.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:34:34.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:34:34.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:34:34.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-15T03:34:34.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-15T03:34:34.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-15T03:34:34.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-15T03:34:34.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-15T03:34:34.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-15T03:34:34.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:35.305 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_WARN There are daemons running an older version of ceph 2024-09-15T03:34:35.305 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2024-09-15T03:34:35.305 INFO:tasks.workunit.client.0.smithi137.stdout: mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-15T03:34:35.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:35.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-15T03:34:35.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:35.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2024-09-15T03:34:36.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:36.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-15T03:34:36.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:153: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.2 2024-09-15T03:34:36.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:34:36.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:34:36.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:34:36.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:34:36.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:34:37.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:34:37.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-15T03:34:37.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: activate_osd td/ver-health 2 2024-09-15T03:34:37.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:34:37.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:34:37.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:34:37.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:34:37.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:34:37.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:37.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:37.052 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:37.052 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:37.052 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:37.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:37.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:34:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-15T03:34:37.054 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:34:37.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:34:37.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:34:37.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-15T03:34:37.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:34:37.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:34:37.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:34:37.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:34:37.071 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:37.070+0000 7fdfa7a20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:34:37.131 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:37.130+0000 7fdfa7a20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:34:37.132 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:37.131+0000 7fdfa7a20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:34:37.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:34:37.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:34:37.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:34:37.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:34:37.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:34:37.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:37.472 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:34:37.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:34:37.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:37.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:37.652 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:37.652+0000 7fdfa7a20440 -1 Falling back to public interface 2024-09-15T03:34:37.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:38.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:38.826 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:34:38.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:38.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:34:38.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:38.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:39.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:40.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:40.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:40.200 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:34:40.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:34:40.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:40.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:40.412 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:40.411+0000 7fdfa7a20440 -1 osd.2 17 log_to_monitors true 2024-09-15T03:34:40.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:41.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:41.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:41.553 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:34:41.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:34:41.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:41.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:34:41.909 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 18 last_clean_interval [15,17) [v2:127.0.0.1:6816/203854,v1:127.0.0.1:6817/203854] [v2:127.0.0.1:6818/203854,v1:127.0.0.1:6819/203854] exists,up d3876208-76ba-4131-8ba0-b7aefffb1653 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:155: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.0 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:34:41.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: activate_osd td/ver-health 0 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:34:42.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:42.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:34:42.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-15T03:34:42.021 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:34:42.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:34:42.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:34:42.022 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-15T03:34:42.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:34:42.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:34:42.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:34:42.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:34:42.039 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:42.037+0000 7f7702153440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:34:42.063 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:42.061+0000 7f7702153440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:34:42.064 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:42.062+0000 7f7702153440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:34:42.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:34:42.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:34:42.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:34:42.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:34:42.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:34:42.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:42.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:34:42.385 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:34:42.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:42.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:34:42.578 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:42.577+0000 7f7702153440 -1 Falling back to public interface 2024-09-15T03:34:42.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:43.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:43.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:43.890 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:34:43.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:34:43.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:43.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:34:44.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:45.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:45.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:45.248 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:34:45.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:34:45.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:45.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:34:45.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:34:45.822 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:34:45.821+0000 7f7702153440 -1 osd.0 19 log_to_monitors true 2024-09-15T03:34:46.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:34:46.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:34:46.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:34:46.606 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-09-15T03:34:46.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:34:46.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:34:47.125 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [5,19) [v2:127.0.0.1:6800/204424,v1:127.0.0.1:6801/204424] [v2:127.0.0.1:6802/204424,v1:127.0.0.1:6803/204424] exists,up c430bee4-4b33-4c19-a688-8e9c774c41b8 2024-09-15T03:34:47.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:34:47.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:34:47.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:34:47.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:158: TEST_check_version_health_2: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:34:47.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:34:47.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:34:48.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:34:48.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:34:48.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:34:48.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:34:49.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:34:50.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:34:50.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:34:50.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:34:50.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:34:50.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-15T03:34:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-15T03:34:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-15T03:34:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-15T03:34:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-15T03:34:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:160: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:51.201 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-15T03:34:51.201 INFO:tasks.workunit.client.0.smithi137.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-15T03:34:51.201 INFO:tasks.workunit.client.0.smithi137.stdout: mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-15T03:34:51.201 INFO:tasks.workunit.client.0.smithi137.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-15T03:34:51.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: ceph health 2024-09-15T03:34:51.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:34:51.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:51.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:34:52.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:52.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-15T03:34:52.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:52.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: grep -q 'mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-15T03:34:53.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: ceph health detail 2024-09-15T03:34:53.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-15T03:34:53.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-15T03:34:53.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:34:53.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:34:53.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:34:53.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:34:53.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:34:53.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:34:53.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:34:53.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:34:53.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:34:53.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:34:53.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:34:53.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:34:53.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:34:53.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:34:53.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:34:53.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:34:53.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:34:53.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:34:53.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:34:53.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:34:53.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:34:53.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:34:53.958 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:34:53.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:53.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:53.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:34:53.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:34:53.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:34:53.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:34:53.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:34:53.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:34:53.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:34:53.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:34:53.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:34:53.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:34:53.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:34:53.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:34:53.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:34:53.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:34:53.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:34:53.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:34:53.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:34:53.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:34:53.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:34:53.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:53.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:53.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:34:53.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:34:53.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:34:53.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-15T03:34:53.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:34:53.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:53.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:53.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T03:34:53.973 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:34:53.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:34:53.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:34:53.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-15T03:34:53.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_3 td/ver-health 2024-09-15T03:34:53.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:170: TEST_check_version_health_3: local dir=td/ver-health 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:173: TEST_check_version_health_3: CEPH_ARGS='--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:175: TEST_check_version_health_3: setup td/ver-health 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:34:53.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:34:53.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:34:53.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:34:53.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:34:53.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:34:53.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:34:53.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:34:53.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:34:53.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:34:53.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:34:53.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:34:53.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:34:53.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:34:53.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:34:53.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:34:53.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:34:53.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:34:53.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:53.981 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:53.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:34:53.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:34:53.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:34:53.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-15T03:34:53.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:34:53.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:53.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:53.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T03:34:53.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:34:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:34:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:34:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-15T03:34:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:178: TEST_check_version_health_3: run_mon td/ver-health a --public-addr=127.0.0.1:7165 2024-09-15T03:34:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-15T03:34:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:34:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:34:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:34:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-15T03:34:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:54.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 2024-09-15T03:34:54.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:34:54.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:34:54.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:34:54.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:34:54.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:34:54.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:34:54.605 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:34:54.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get fsid 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:54.681 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:54.682 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T03:34:54.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:34:54.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get mon_host 2024-09-15T03:34:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:179: TEST_check_version_health_3: run_mon td/ver-health b --public-addr=127.0.0.1:7166 2024-09-15T03:34:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-15T03:34:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:34:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-15T03:34:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:34:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-15T03:34:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 2024-09-15T03:34:55.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:34:55.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:55.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:55.351 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:55.351 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:55.352 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:55.352 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:55.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 2024-09-15T03:34:55.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:34:55.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:34:55.739 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:34:55.739 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:55.739 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:55.739 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:34:55.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:34:55.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get fsid 2024-09-15T03:34:55.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-15T03:34:55.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.b.asok 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:34:55.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.b.asok config get mon_host 2024-09-15T03:34:55.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:181: TEST_check_version_health_3: local start_osd_time=281 2024-09-15T03:34:55.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-15T03:34:55.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: run_osd td/ver-health 0 2024-09-15T03:34:55.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:34:55.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-15T03:34:55.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:34:55.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9cf9a1dc-853f-4ca5-9718-ed8135ae0bda 2024-09-15T03:34:55.899 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 9cf9a1dc-853f-4ca5-9718-ed8135ae0bda 2024-09-15T03:34:55.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9cf9a1dc-853f-4ca5-9718-ed8135ae0bda' 2024-09-15T03:34:55.900 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:34:55.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfVeZmrg4zNhAAjMWnCWtD4BYs2ag4RU/R1w== 2024-09-15T03:34:55.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfVeZmrg4zNhAAjMWnCWtD4BYs2ag4RU/R1w=="}' 2024-09-15T03:34:55.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9cf9a1dc-853f-4ca5-9718-ed8135ae0bda -i td/ver-health/0/new.json 2024-09-15T03:35:02.212 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:35:02.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-15T03:35:02.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfVeZmrg4zNhAAjMWnCWtD4BYs2ag4RU/R1w== --osd-uuid 9cf9a1dc-853f-4ca5-9718-ed8135ae0bda 2024-09-15T03:35:02.246 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:02.245+0000 7f837a24f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:02.248 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:02.247+0000 7f837a24f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:02.249 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:02.248+0000 7f837a24f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:02.346 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:02.344+0000 7f837a24f440 -1 memstore(td/ver-health/0) td/ver-health/0 2024-09-15T03:35:02.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-15T03:35:02.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:35:02.862 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-09-15T03:35:02.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:35:02.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:35:03.047 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:35:03.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:35:03.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:35:03.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:35:03.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:35:03.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:35:03.070 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:03.068+0000 7f41d4d24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:03.084 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:03.082+0000 7f41d4d24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:03.085 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:03.083+0000 7f41d4d24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:03.086 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:03.084+0000 7f41d4d24440 -1 Falling back to public interface 2024-09-15T03:35:03.095 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:03.094+0000 7f41d4d24440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:35:03.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:35:03.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:35:03.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:35:03.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:35:03.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:35:03.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:03.305 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:35:03.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:35:03.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:03.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:35:03.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:35:04.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:35:04.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:04.607 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:35:04.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:35:04.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:04.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:35:04.827 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/205529,v1:127.0.0.1:6801/205529] [v2:127.0.0.1:6802/205529,v1:127.0.0.1:6803/205529] exists,up 9cf9a1dc-853f-4ca5-9718-ed8135ae0bda 2024-09-15T03:35:04.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:35:04.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:35:04.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:35:04.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: run_osd td/ver-health 1 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-15T03:35:04.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:35:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:35:04.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b0bca5f-6acb-424f-9ec5-e952aea0240e 2024-09-15T03:35:04.834 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 4b0bca5f-6acb-424f-9ec5-e952aea0240e 2024-09-15T03:35:04.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4b0bca5f-6acb-424f-9ec5-e952aea0240e' 2024-09-15T03:35:04.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:35:04.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoVeZmbWNFMhAAOn6kBFxGHTMEae9iZ8Bxng== 2024-09-15T03:35:04.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoVeZmbWNFMhAAOn6kBFxGHTMEae9iZ8Bxng=="}' 2024-09-15T03:35:04.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b0bca5f-6acb-424f-9ec5-e952aea0240e -i td/ver-health/1/new.json 2024-09-15T03:35:05.153 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:35:05.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-15T03:35:05.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoVeZmbWNFMhAAOn6kBFxGHTMEae9iZ8Bxng== --osd-uuid 4b0bca5f-6acb-424f-9ec5-e952aea0240e 2024-09-15T03:35:05.188 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:05.186+0000 7ff04c40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:05.189 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:05.188+0000 7ff04c40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:05.190 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:05.189+0000 7ff04c40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:05.263 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:05.261+0000 7ff04c40c440 -1 memstore(td/ver-health/1) td/ver-health/1 2024-09-15T03:35:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-15T03:35:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:35:05.813 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-09-15T03:35:05.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:35:05.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:35:05.996 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:35:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:35:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:35:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:35:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:35:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:35:06.019 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:06.017+0000 7ffa5e721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:06.020 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:06.019+0000 7ffa5e721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:06.021 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:06.020+0000 7ffa5e721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:06.023 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:06.021+0000 7ffa5e721440 -1 Falling back to public interface 2024-09-15T03:35:06.032 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:06.031+0000 7ffa5e721440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:35:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:35:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:35:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:35:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:35:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:35:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:06.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:35:06.372 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:35:06.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:06.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:35:06.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:35:07.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:35:07.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:07.575 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:35:07.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:35:07.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:07.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:35:07.750 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/205734,v1:127.0.0.1:6809/205734] [v2:127.0.0.1:6810/205734,v1:127.0.0.1:6811/205734] exists,up 4b0bca5f-6acb-424f-9ec5-e952aea0240e 2024-09-15T03:35:07.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:35:07.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:35:07.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:35:07.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-15T03:35:07.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: run_osd td/ver-health 2 2024-09-15T03:35:07.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-15T03:35:07.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:35:07.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:35:07.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:35:07.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-15T03:35:07.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:35:07.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:35:07.756 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:35:07.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:35:07.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-15T03:35:07.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:35:07.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1ef63862-991f-4586-95a3-87f193283059 2024-09-15T03:35:07.758 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 1ef63862-991f-4586-95a3-87f193283059 2024-09-15T03:35:07.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1ef63862-991f-4586-95a3-87f193283059' 2024-09-15T03:35:07.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:35:07.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrVeZmMT24LRAA8PBFQJC8erEkN0TeIu5FwA== 2024-09-15T03:35:07.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrVeZmMT24LRAA8PBFQJC8erEkN0TeIu5FwA=="}' 2024-09-15T03:35:07.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1ef63862-991f-4586-95a3-87f193283059 -i td/ver-health/2/new.json 2024-09-15T03:35:08.506 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-09-15T03:35:08.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-15T03:35:08.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrVeZmMT24LRAA8PBFQJC8erEkN0TeIu5FwA== --osd-uuid 1ef63862-991f-4586-95a3-87f193283059 2024-09-15T03:35:08.541 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:08.540+0000 7fb479e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:08.543 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:08.542+0000 7fb479e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:08.544 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:08.543+0000 7fb479e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:08.633 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:08.632+0000 7fb479e0c440 -1 memstore(td/ver-health/2) td/ver-health/2 2024-09-15T03:35:09.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-15T03:35:09.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:35:09.256 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-09-15T03:35:09.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:35:09.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:35:09.444 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:35:09.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:35:09.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:35:09.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:35:09.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:35:09.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:35:09.467 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:09.465+0000 7f7c32b2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:09.479 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:09.477+0000 7f7c32b2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:09.480 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:09.478+0000 7f7c32b2b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:09.481 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:09.479+0000 7f7c32b2b440 -1 Falling back to public interface 2024-09-15T03:35:09.491 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:09.490+0000 7f7c32b2b440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:35:09.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:35:09.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:35:09.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:35:09.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:35:09.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:35:09.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:09.684 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:35:09.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:35:09.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:09.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:35:09.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:35:10.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:35:10.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:10.901 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:35:10.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:35:10.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:10.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:35:11.079 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/205939,v1:127.0.0.1:6817/205939] [v2:127.0.0.1:6818/205939,v1:127.0.0.1:6819/205939] exists,up 1ef63862-991f-4586-95a3-87f193283059 2024-09-15T03:35:11.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:35:11.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:35:11.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:35:11.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:187: TEST_check_version_health_3: local warn_older_version_delay=36 2024-09-15T03:35:11.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:189: TEST_check_version_health_3: sleep 5 2024-09-15T03:35:16.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:190: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:16.261 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_OK 2024-09-15T03:35:16.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:16.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-15T03:35:16.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:193: TEST_check_version_health_3: ceph tell 'mon.*' injectargs '--mon_warn_older_version_delay 36' 2024-09-15T03:35:16.611 INFO:tasks.workunit.client.0.smithi137.stdout:mon.a: {} 2024-09-15T03:35:16.641 INFO:tasks.workunit.client.0.smithi137.stdout:mon.b: {} 2024-09-15T03:35:16.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:194: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.1 2024-09-15T03:35:16.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:35:16.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:35:16.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:35:16.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:35:16.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:35:16.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:35:16.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-15T03:35:16.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-15T03:35:16.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: activate_osd td/ver-health 1 2024-09-15T03:35:16.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:35:16.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:35:16.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:35:16.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:35:16.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:35:16.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:35:16.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:35:16.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:35:16.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-15T03:35:16.762 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-09-15T03:35:16.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:35:16.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:35:16.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-15T03:35:16.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:35:16.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:35:16.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:35:16.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:35:16.779 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:16.779+0000 7f71d2f35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:16.821 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:16.821+0000 7f71d2f35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:16.823 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:16.822+0000 7f71d2f35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:16.823 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:16.823+0000 7f71d2f35440 -1 Falling back to public interface 2024-09-15T03:35:16.833 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:16.833+0000 7f71d2f35440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:35:17.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:35:17.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:35:17.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:35:17.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:35:17.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:35:17.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:17.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:35:17.033 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:35:17.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:17.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:35:17.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:35:18.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:35:18.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:18.337 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:35:18.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:35:18.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:18.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:35:18.509 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6808/206163,v1:127.0.0.1:6809/206163] [v2:127.0.0.1:6810/206163,v1:127.0.0.1:6811/206163] exists,up 4b0bca5f-6acb-424f-9ec5-e952aea0240e 2024-09-15T03:35:18.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:35:18.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:35:18.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:35:18.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:200: TEST_check_version_health_3: sleep 10 2024-09-15T03:35:28.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:28.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:205: TEST_check_version_health_3: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 20 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:28.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:28.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:29.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:29.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:29.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:29.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:30.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:31.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:31.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:31.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:31.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:31.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:32.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:33.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:33.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:33.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:33.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:33.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:34.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:34.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:34.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:34.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:34.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:35.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:35.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:35.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:35.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:36.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:37.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:37.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:37.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:37.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:37.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:38.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:38.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:38.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:38.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:38.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:39.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:39.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:39.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:39.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:39.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:40.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:40.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:40.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:40.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:40.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-15T03:35:40.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-15T03:35:40.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-15T03:35:40.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-15T03:35:40.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-15T03:35:40.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:207: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:41.039 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2024-09-15T03:35:41.040 INFO:tasks.workunit.client.0.smithi137.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-15T03:35:41.040 INFO:tasks.workunit.client.0.smithi137.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-15T03:35:41.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:41.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-15T03:35:41.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:41.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-15T03:35:41.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:41.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-15T03:35:41.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:213: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.2 2024-09-15T03:35:41.645 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:35:41.645 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:35:41.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:35:41.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:35:41.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:35:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:35:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-15T03:35:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: activate_osd td/ver-health 2 2024-09-15T03:35:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:35:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:35:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:35:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:35:41.754 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:35:41.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:35:41.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:35:41.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:35:41.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:35:41.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-15T03:35:41.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:35:41.756 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-09-15T03:35:41.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:35:41.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-15T03:35:41.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:35:41.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:35:41.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:35:41.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:35:41.774 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:41.773+0000 7fa43a834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:41.776 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:41.775+0000 7fa43a834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:41.777 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:41.776+0000 7fa43a834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:41.778 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:41.777+0000 7fa43a834440 -1 Falling back to public interface 2024-09-15T03:35:41.788 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:41.787+0000 7fa43a834440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:35:42.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:35:42.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:35:42.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:35:42.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:35:42.107 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:35:42.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:35:42.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:42.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:35:42.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:42.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:35:42.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:35:43.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:35:43.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:43.370 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:35:43.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:35:43.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:43.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:35:43.546 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [0,0) [v2:127.0.0.1:6816/206704,v1:127.0.0.1:6817/206704] [v2:127.0.0.1:6818/206704,v1:127.0.0.1:6819/206704] exists,up 1ef63862-991f-4586-95a3-87f193283059 2024-09-15T03:35:43.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:35:43.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:35:43.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:35:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:215: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.0 2024-09-15T03:35:43.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:35:43.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:35:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:35:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:35:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:35:43.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:35:43.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-15T03:35:43.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: activate_osd td/ver-health 0 2024-09-15T03:35:43.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:35:43.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:35:43.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:35:43.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:35:43.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:35:43.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:35:43.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-15T03:35:43.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:35:43.658 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-09-15T03:35:43.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d1b060d4-12e1-48bf-a3e9-ee40353cd10a --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:35:43.659 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-15T03:35:43.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:35:43.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:35:43.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:35:43.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:35:43.676 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:43.675+0000 7f23b3135440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:43.756 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:43.755+0000 7f23b3135440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:43.757 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:43.756+0000 7f23b3135440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:35:43.758 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:43.757+0000 7f23b3135440 -1 Falling back to public interface 2024-09-15T03:35:43.769 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-15T03:35:43.767+0000 7f23b3135440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:35:43.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:35:43.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:35:43.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:35:43.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:35:43.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:35:43.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:43.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:35:43.928 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-09-15T03:35:43.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:43.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:35:44.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:35:45.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:35:45.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:35:45.581 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-09-15T03:35:45.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:35:45.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:35:45.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:35:45.765 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 26 up_thru 0 down_at 24 last_clean_interval [0,0) [v2:127.0.0.1:6800/206869,v1:127.0.0.1:6801/206869] [v2:127.0.0.1:6802/206869,v1:127.0.0.1:6803/206869] exists,up 9cf9a1dc-853f-4ca5-9718-ed8135ae0bda 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:218: TEST_check_version_health_3: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:45.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:45.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:46.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:46.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:46.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:46.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:47.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:48.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:48.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:48.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:48.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:48.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:49.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:49.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:49.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:49.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:49.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-15T03:35:50.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-15T03:35:50.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-15T03:35:50.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-15T03:35:50.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:50.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-15T03:35:50.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-15T03:35:50.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-15T03:35:50.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-15T03:35:50.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-15T03:35:50.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:220: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:50.972 INFO:tasks.workunit.client.0.smithi137.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-15T03:35:50.972 INFO:tasks.workunit.client.0.smithi137.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-15T03:35:50.972 INFO:tasks.workunit.client.0.smithi137.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-15T03:35:50.973 INFO:tasks.workunit.client.0.smithi137.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-15T03:35:50.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:50.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-15T03:35:51.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:51.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-15T03:35:51.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:51.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-15T03:35:51.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: ceph health detail 2024-09-15T03:35:51.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-15T03:35:51.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-15T03:35:51.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:35:51.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:35:51.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:35:51.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:35:51.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:35:51.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:35:51.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:35:51.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:35:51.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:35:51.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:35:51.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:35:51.847 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:35:51.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:35:51.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:35:51.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:35:51.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:35:51.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:35:51.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:35:51.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:35:51.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:35:51.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:35:51.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:35:51.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:35:51.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:35:51.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:35:51.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:35:51.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/ver-health 0 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:35:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:35:51.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:35:51.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:35:51.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:35:51.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:35:51.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:35:51.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:35:51.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:35:51.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:35:51.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:35:51.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:35:51.864 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:35:51.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:35:51.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T03:35:51.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-15T03:35:51.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:35:51.866 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:35:51.867 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T03:35:51.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T03:35:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:35:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:35:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T03:35:51.868 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:35:51.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:35:51.933 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-09-15T03:35:51.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T03:35:52.238 DEBUG:teuthology.parallel:result is None 2024-09-15T03:35:52.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:35:52.263 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:35:52.263 DEBUG:teuthology.orchestra.run.smithi137:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:35:52.328 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:35:52.329 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T03:35:52.338 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T03:35:52.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:35:52.414 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T03:35:52.422 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T03:35:52.422 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:35:52.447 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:35:52.452 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:35:52.453 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-09-15T03:35:52.453 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +202us[ +198us] +/- 44ms 2024-09-15T03:35:52.453 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 110 -185us[ -194us] +/- 58ms 2024-09-15T03:35:52.453 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 -119us[ -119us] +/- 70ms 2024-09-15T03:35:52.453 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:35:52.453 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T03:35:52.462 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T03:35:52.462 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T03:35:52.493 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:35:52.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:35:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:35:52.579 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T03:35:52.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T03:35:52.618 INFO:teuthology.task.internal:Duration was 3765.455307 seconds 2024-09-15T03:35:52.619 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T03:35:52.660 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T03:35:52.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:35:52.699 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:35:52.894 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T03:35:52.894 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-09-15T03:35:52.894 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:35:52.957 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T03:35:52.957 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:35:53.023 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T03:35:53.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:35:53.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T03:35:53.169 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T03:35:53.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:35:53.194 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T03:35:53.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:35:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-09-15T03:35:53.271 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:35:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:35:53.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T03:35:53.335 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T03:35:53.337 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905167/remote/smithi137 2024-09-15T03:35:53.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:35:53.423 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T03:35:53.423 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:35:53.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T03:35:53.472 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T03:35:53.472 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T03:35:53.546 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T03:35:53.546 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:35:53.561 INFO:teuthology.orchestra.run.smithi137.stdout: 265811 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:35 /home/ubuntu/cephtest 2024-09-15T03:35:53.562 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T03:35:53.588 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T03:35:53.672 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 3765.455306529999 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T03:35:53.672 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:35:53.760 INFO:teuthology.run:pass